Notes from the compute layer

Apache Iceberg, Delta Lake, and Apache Hudi solve similar lakehouse problems but suit different workloads. Iceberg is the strongest choice for open, multi-engine analytics; Delta Lake fits Databricks and Spark-centric environments; and Hudi is designed for streaming, CDC, and frequent upserts. This guide compares their metadata architectures, transaction models, concurrency, schema evolution, partitioning, and row-level mutations, while explaining how a format-agnostic engine can query all three without data movement or pipeline rewrites.

This blog shows how e6data decomposes metric anomalies into structural, causal, and narrative layers and feeds the output into an AI agent that can autonomously investigate incidents.

Learn proven data warehouse cost optimization strategies for Snowflake, BigQuery, Redshift, and more. This 2026 guide covers pricing models, cost drivers, step-by-step optimization tactics, and how e6data's Atomic Architecture delivers up to 60% lower compute costs without changing your architecture.

This post explains a subtle async Rust bug where tokio::sync::Mutex appears to deadlock despite logs showing the lock was released. The real issue is a wrapper future that stops polling its inner future after wake-up, violating the waker contract and trapping the semaphore permit inside Tokio’s waiter queue. The article walks through the reproducer, reframes why the Coffman conditions seem misleading here, and ends with practical fixes: pause or cancel work only at safe boundaries, or refactor critical sections to avoid holding async locks across .await.

Apache Iceberg can OOM during planning when the file count inflates the metadata to terabytes. Fix it by streaming metadata reads (bounded batches), layered pruning (manifest-list → file-level), and treating metadata as queryable data to push down filters and project only needed stats—turning minutes or failures into seconds.

The Variant data type stores semi-structured JSON in a binary, self-describing format with a per-row key dictionary (“metadata”) and compact values. Engines can jump directly to fields by path, avoiding full JSON parses. The result is flexible schema-on-read with significantly faster queries than plain JSON strings.

e6data’s blog explains how German Strings improve string performance in real-time analytics by replacing pointer-heavy strings with a compact 16-byte string struct. The layout stores length and an inline string prefix, so equality checks, joins, and prefix comparisons short-circuit on those bytes, boosting cache locality and reducing pointer dereferences. We show how Arrow’s StringView represents short strings fully inlined and long strings via buffer index and offset, enabling fewer copies, reuse of repeated values, and faster ingestion from Parquet. The post details benefits, trade-offs, and our implementation results: up to 3 to 3.5x speedups in microbenchmarks and consistent gains in string-heavy workloads. Readers learn when the optimization shines, how immutable strings and columnar processing help, and where adoption appears in Apache Arrow and DataFusion. If you care about query latency, throughput, and memory efficiency, this walkthrough gives a practical path to making text behave more like numbers with German Strings.

Partition projection replaces manual, Hive-style partition registration with rule-based metadata that query engines compute on the fly. You keep the benefits of partition pruning while eliminating crawler/repair jobs and bloated catalog metadata, which cuts ETL costs and speeds up queries across cloud-agnostic data lakehouse platforms.

This is a guide to building a lakehouse pipeline in Snowflake: Snowpipe auto‑ingests JSON orders to a landing table; a stored procedure cleans and transforms data into curated core tables, then writes to Snowflake‑managed Apache Iceberg tables stored as Parquet. We demonstrate how daily tasks trigger the procedure at 02:00 UTC and run sync checks, comprising row counts, set differences, and optional checksums, to ensure parity between core tables and Iceberg snapshots. The design combines Snowflake governance, SQL simplicity, and ACID performance with Iceberg’s open catalog, enabling multi‑engine analytics while avoiding vendor lock‑in.

How to implement detection of subgraphs within a query plan | Demonstrate pruning and cost-based decisions | Redundant computations | Automatic CTE

This blog compares the top embedded analytics tools for B2B SaaS companies in 2026, covering platforms such as e6data, Looker, Tableau, Power BI, Sigma, ThoughtSpot, Sisense, GoodData, Qrvey, Metabase, Superset, and Mode. It explains what SaaS teams should evaluate when choosing an embedded analytics platform, including embedding flexibility, API-first architecture, multi-tenant support, row-level security, scalability, and white-label customization. The article positions e6data as the best fit for high-performance analytics on large datasets, while also outlining where other platforms are stronger, such as governance, visualization, search, open-source flexibility, or rapid deployment.

Run semantic search in SQL with cosine similarity and without a separate vector DB. e6data’s LOAD VECTOR and Blitz handle on-the-fly embeddings, quantized pruning, and precise reranking. The post compares MiniLM, SciBERT, CodeBERT, and shares quick tips on thresholds Vs. top-K for accurate, sub-second results.

The emerging Iceberg catalogs are reshaping metadata management strategies for 2025. From Git-style branching in Nessie to the lean Rust performance of Lakekeeper, a new wave of emerging catalogs: Nessie, Polaris, Gravitino, Lakekeeper, and Unity Catalog, are challenging the long-entrenched Hadoop, Hive, and Glue stores. Each promises cross-table transactions, fine-grained governance, and multi-engine freedom, answering developers’ calls for faster, safer lakehouse operations. DuckDB’s DuckLake shows how even SQL engines are entering the arena, while Apache Iceberg’s REST spec keeps everyone talking the same protocol. Whether you crave geo-distributed metadata, zero-copy isolation, or a single-binary deploy, the catalog war is far from over. And the winners will define how data engineers build lakehouses for years to come.

Connect once, query everywhere. Our latest release provides advanced support for Apache Iceberg, Apache Polaris, Apache Hudi, and Delta Lake, enabling you to run fast SQL queries across every open table format without copying data or rewriting pipelines.

Traditional multi-statement SQL User-Defined Functions give developers modular, expressive power but at the cost of brutal row-by-row execution and optimizer blindness. Drawing on Microsoft’s Froid research, e6data dissolves that performance tax: its new engine automatically inlines procedural UDF logic back into the set-oriented query plan, eliminating context switches and unleashing parallelism. The post explains why UDFs matter, dissects the pain points of inline vs. multi-statement functions, and shows how e6data fuses loops, variables, and branching logic into a single relational plan. The result is orders-of-magnitude speed-ups with no code changes, setting the stage for a deeper dive into the inlining algorithms in upcoming articles.

Geospatial performance bottlenecks appear the instant a lat/lon B-tree creeps past a hundred million rows. This guide benchmarks H3 hex grids against Quadkey square tiles, shows how to hex-bin heat-maps, shard warehouses, and even stream 3-D voxel tiles on GPUs.

Vector and semantic search let lakehouse teams unlock 80% of enterprise data that lives in free-form text, images, and other unstructured formats. By embedding high-dimensional vectors directly into the SQL optimizer, e6data’s “Unify, Don’t Migrate” engine runs cosine-similarity joins faster without extra ETL or a separate vector database. The result: lower costs, simpler pipelines, and far richer insights from customer voice, support chats, docs, and more.

e6data brings SQL and AI together, letting you query unstructured data directly in Microsoft Fabric 10x faster and 60% lower cost—no pipelines, no workarounds, just fast answers.

e6data integrates with Microsoft Fabric to make querying data about 33% faster (as per benchmark reports) — helping you speed up data analytics while cutting costs by 2-3x. It directly taps into your OneLake data without any data movement.

Query your existing OneLake data about 33% faster, cutting analytics costs by 2-3x without changing your Fabric workflow.

e6data's founding engineer and Head of Engineering, Sudarshan Lakshminarasimhan, unpacks the internals of their compute engine on Zero Prime podcast

e6data’s hybrid lakehouse keeps compute beside the data across AWS, Azure, GCP, and on-prem, wiping out cross-region egress charges and reducing query times by up to 10x. A location-aware scheduler plus open table formats (Parquet, Iceberg, Delta) avoids vendor lock-in, while a single IAM hook streamlines governance.

This article explains why enterprises must own the context layer for AI that gives AI agents reliable business meaning. It shows how an enterprise context layer connects semantic metadata, governance, lineage, permissions, institutional knowledge, ontologies, and an enterprise knowledge graph. It also clarifies context layer vs semantic layer and semantic layer vs ontology, arguing that portable data does not guarantee portable intelligence or vendor-independent understanding. The recommendations reflect the article’s definition of a context layer as the connected business definitions, semantic metadata, relationships, lineage, permissions, freshness signals, and institutional knowledge needed to interpret enterprise data correctly. They also preserve its central argument: enterprises should retain control of this intelligence independently of individual tools, engines, and vendors.