Artificial Intelligence

AIOpen Source

Black Forest Labs launches FLUX 3 capable of generating images and 20-second video with audio — but in limited release to start

Black Forest Labs (BFL) is expanding its FLUX family beyond image generation with today's launch of FLUX 3 , a multimodal frontier model trained to understand and generate images, or combined audio/video clips up to 20 seconds from a single prompt — and to extend the same underlying architecture to robotic vision and actions. The Freiburg, Germany-based AI lab says FLUX 3 is jointly trained across those modalities rather than assembling separate image, video and audio models behind a common interface. That distinction is central to the company's pitch: BFL wants enterprises to think about creative generation, simulation, computer use and robotics as connected applications of a single capability it calls visual intelligence — models, in the company's words, "that can perceive, predict, and act across physical and digital environments." This release marks BFL's first public video generation model. FLUX 3 will be offered through four product lines: FLUX 3 Video, FLUX 3 Image, FLUX 3 Action and the upcoming, open source FLUX 3 Dev. FLUX 3 Video, with optional native audio generation, and FLUX 3 Action are entering a gated "Early Access" program now , to which anyone can apply, but which BFL must approve. There is presently no public access through BFL's application programming interface (API) or those of partners yet, but the company says FLUX 3 Image will roll out in the coming weeks, followed by general availability. The limited initial availability rollout echoes the release strategies of new models from other frontier labs in the U.S. lately, including Anthropic and OpenAI , though those were ostensibly for security concerns and due to government request. What the company has not announced is pricing, production service-level commitments, evaluation methodology, sample sizes, rater counts or any image-model benchmarks at all. Enterprise buyers therefore cannot yet calculate total cost of ownership or independently reproduce the video comparisons. Another big notable omission: FLUX 3 is not launching with downloadable weights at this time, nor an open source license. BFL says faster and open-weight versions will arrive later this year, and its technical blog names FLUX 3 Dev as "open-weight access to a multimodal backbone, for content creation (video, audio and image) and action prediction" — a considerably broader commitment than any previous FLUX Dev release, all of which covered images only. But it arrives last in the sequence. Developers accustomed to receiving a locally deployable FLUX variant alongside — or soon after — a major model announcement will have to wait. That delay does not negate the company's commitment, but it is disappointing given the role open weights have played in FLUX's adoption thus far. Flux 3 is rated higher than the competition, but missing pricing and benchmarking details may prevent rapid enterprise adoption BFL has published several benchmark comparisons, but they're qualified as preliminary — with full benchmark results and methodology to be published later during broader general availability. In early head-to-head preference testing on 10-second, 720p text-to-video clips with audio, the company says FLUX 3 was preferred over Luma Ray 3.2 in 93% of comparisons, Runway Gen-4.5 in 77%, Grok Imagine Video in 69%, Kling v3 Pro in 60%, Happy Horse v1 in 59%, Happy Horse 1.1 in 57%, and both Seedance 2.0 and Google's Gemini Omni Flash in 52%. One caveat travels with every one of those figures, and it comes from BFL itself. The chart carrying the results is labeled a "preliminary evaluation of an early FLUX 3 candidate" — meaning the numbers describe a pre-release checkpoint rather than the model now entering early access. That cuts both ways: the shipping model may perform better, but nothing published today measures what customers will actually call. Luma Ray 3.2 and Runway Gen-4.5, where FLUX 3 posted 93% and 77%, are the softest comparisons on the list — established products, but not the models currently setting the pace in independent video rankings. Those are real wins, and they are the ones least likely to change an enterprise shortlist. Seedance 2.0, at 52%, is a statistical coin flip against a model most Western enterprises cannot currently procure. ByteDance indefinitely postponed Seedance 2.0's international rollout after Netflix, Warner Bros., Disney, Paramount and Sony sent legal threats over alleged systematic copyright infringement, and that suspension remains in place. Tying a frozen product is neither a strong claim nor a damaging one. Gemini Omni Flash , also at 52%, matters much more. Omni is the closest large-platform analogue to what FLUX 3 is attempting — multimodal input, video and audio-aware creation, conversational editing — and by BFL's own measurement, the two are indistinguishable on 10-second text-to-video quality. Google's advantage in that matchup is that Omni is generally available via Google's Gemini API for $0.10 per second of generated 720p video, or a 10-second clip for around. One regional wrinkle matters for a German company's home market. Editing uploaded video is unavailable to Omni Flash users in the European Economic Area, Switzerland and the United Kingdom, though editing video the model itself generated is permitted. A European enterprise that wants to run its existing footage through a generative editing pass cannot currently do so on Omni Flash. Here's a rough guide for enterprises considering which video models to rely upon: Model Max single-generation duration Max resolution Key constraints Price per 10-second clip (720p) Price per 10-second clip (1080p) Price per 10-second clip (4K) FLUX 3 Video 20 seconds Not stated; evaluations run at 720p Early access; no published SLA or pricing Not announced Not announced Not announced HappyHorse 1.1 15 seconds 1080p No 4K; closed weights Not published (v1.0 reseller rate is ~$1.82) Not published (v1.0 reseller rate is ~$3.12) n/a Veo 3.1 Per-second billing 4K Supports clip extension; preview $4.00 $4.00 $6.00 Veo 3.1 Fast Per-second billing 4K Preview $1.00 $1.20 $3.00 Veo 3.1 Lite Per-second billing 1080p No 4K, no clip extension; preview $0.50 $0.80 n/a Gemini Omni Flash 10 seconds (3s minimum) 720p at 24 FPS Preview abd no EU access $1.00 n/a n/a One architecture for media generation and physical action FLUX 3 builds on Self-Flow , BFL's method for aligning multimodal understanding and generation within one architecture, publicized back in March 2026. The company says it significantly scaled up compute and data to train across video, images and audio simultaneously, and that testing showed video generation and action prediction do not require separate foundations — the same architecture could be extended to action prediction without sacrificing what it learned from video. "We place vision at the center of our approach because it is the most signal-rich medium of the physical world. Images convey structure, images and video teach spatial relationships, video teaches dynamics, and actions reveal causal relationships. But vision alone is not the complete picture," said Robin Rombach, co-founder and CEO of BFL, in a pre-release statement provided to VentureBeat. "True intelligence means perceiving the world: predicting how it will change, taking action, and learning from the results. Joint training within one unified architecture is what will get us there, because each training modality strengthens the others. Audio conveys timing, prosody, and physical events that elude vision. Language conveys goals, abstractions, and instructions that pixels cannot easily express." He put the case more bluntly elsewhere in the announcement: "You can't cheat reality. A model that only learns images can only generate images. But the world is not made of still frames. It moves, sounds, changes, and responds." BFL says FLUX 3 targets creative tooling, media, design, e-commerce and physical AI, supporting video generation with synchronized audio, precise image editing, product and material consistency across motion, multilingual generation and robotic action prediction. It is already being tested by Canva, Burda, Magnific (formerly Freepik), Krea and Picsart. For creative software companies, the appeal is consolidation. A single foundation could potentially support storyboarding, image editing, product rendering, video variation and localization without repeatedly translating assets and instructions between disconnected models. For robotics teams, the potential value is data efficiency. Models that already encode motion, object behavior and physical change may need less task-specific robot training than systems starting from raw demonstrations. What FLUX 3 Video can actually do The video tier is the most concretely specified part of the launch, and it settles a question that had been circulating as rumor: FLUX 3 generates clips of up to 20 seconds with audio in a single generation. Every video output comes with native audio. For comparison, HappyHorse 1.0 tops out at 15 seconds of 1080p with synchronized audio — though BFL has not stated what resolution its 20-second clips run at, and its published evaluations were conducted at 720p. Still, a 20-second long clip from a single prompt is among the longest yet achieved, matching OpenAI's discontinued Sora model. The capability list BFL published covers: Text-to-video generation. Image-to-video generation, either animating from a starting frame or using images as visual references. Video-to-video generation from a reference clip, carrying elements such as a specific character into a new scene or context. Generative video-audio continuation from existing video and audio input. Keyframe-to-video generation for controlled transitions between defined moments. Multilingual dialogue. A broad range of visual styles and aspect ratios, from candid camcorder footage to animation and cinematics. Typography generation and animated design. Agentic chaining of individual clips into longer, multi-shot sequences. That last item is the one enterprise video teams should look at hardest. BFL claims the capabilities combine to produce sequences lasting several minutes, with visual references keeping characters consistent across scenes. If that holds up under production conditions, it addresses the constraint that has kept generative video out of most commercial pipelines: not clip quality, but continuity across shots. It is also the capability where competition is most direct. HappyHorse 1.1's headline upgrade is R2V, or Reference-to-Video, which accepts multiple character reference images to hold identity stable across generated footage — the same problem, approached at the input layer rather than through agentic clip chaining. Alibaba also claims zero-drift lip sync and has specifically targeted the artifacts that mark commercial AI video as synthetic, including facial oiliness and over-sharpening. Character consistency is where this category is being contested, and both companies know it. BFL says FLUX 3 Video is already particularly strong at human facial expressions, associating sounds with physical events, and multilingual output. On the image side, the company says preliminary evaluations conducted during midtraining show significant improvement over earlier FLUX versions in complex prompt handling and text generation, including high-accuracy text in multiple languages. It published no image benchmarks or win rates. FLUX-mimic tests whether video models can become robot models BFL is applying its unified-architecture thesis through FLUX-mimic, a video-action model built on FLUX 3 and developed with Swiss firm Mimic Robotics, one of the first partners to receive early access. The technical blog describes two distinct routes to action prediction: integrating native action prediction directly into FLUX 3, scaling up the initial Self-Flow work; and using the pretrained video backbone as a dynamics-aware foundation from which specialized action models can be finetuned with limited task-specific data. FLUX-mimic is the second route — the FLUX 3 backbone combined with mimic's robot-learning and production-deployment expertise in dexterous manipulation. FLUX-mimic is designed for general-purpose robotic manipulation: helping robots understand a visual scene, predict the consequences of an action, and adapt to new tasks with far less task-specific data. BFL and Mimic Robotics say that depending on task difficulty, the model can be finetuned for a specific manipulation task with as little as 30 minutes of robot data, where prior approaches have required 30 or more hours. "The hardest part of robotics is data," said Elvis Nava, CTO of Mimic Robotics, in a statement provided to VentureBeat. "Every new task normally means hours of a robot repeating itself. Because FLUX-mimic is built on top of frontier video models that already understand how the physical world behaves, it picks up a new task in minutes, not days. This way, we can leapfrog the current state of the art in robot learning." BFL argues that a model trained only on images cannot understand a world that "moves, sounds, changes, and responds," and that physical understanding is what produces convincing generated footage. Google makes a nearly identical claim for Gemini Omni. Its developer documentation cites "world knowledge" that combines "an understanding of physics" with Gemini's grasp of history, science and cultural context. Its marketing is blunter still: "Most AI models just predict the next pixel to build a narrative or an image. Gemini Omni is different," the company posted in June, crediting the model with "an intuitive understanding of forces like gravity, kinetic energy, and fluid dynamics for more realistic movements that follow real-world logic." The practical consequence for enterprise buyers is that world-model language is not a differentiator. Two of the three leading video systems now market physical understanding as their central advantage, and neither has published a benchmark that measures it. There is no standard test for whether generated water behaves like water, whether a dropped object falls at a plausible rate, or whether a sound arrives when the impact does. Human preference ratings capture some of it indirectly. Nothing else on offer captures it at all. Open weights helped make FLUX an industry standard BFL officially launched in summer 2024 and gained a name for itself in the AI industry in the intervening two years for its commitment to open sourcing high-quality AI image models beloved by developers, creatives, and enterprises. The company's founders, including Rombach, Andreas Blattmann and Patrick Esser, previously helped create VQGAN, latent diffusion and Stable Diffusion , the latter the open source technology that kicked off broad AI generation capabilities for the masses and currently used by many AI image generators and companies. That reach translated into commercial distribution. FLUX models now power generative features inside Adobe Photoshop, Picsart and Nous Research's Hermes Agent, among other platforms, and the company cites film director Martin Scorsese among professional users. Wired magazine described Black Forest Labs as a relatively small company that nevertheless became a leading competitor to Silicon Valley's largest AI labs, with FLUX models ranking near the top of image benchmarks and becoming some of the most downloaded text-to-image models on AI code sharing community Hugging Face. The company says it now runs a 100-person team across Freiburg and San Francisco. FLUX.1 Dev, FLUX.1 Kontext Dev, FLUX.1 Fill Dev and related control models, released shortly after the firm's launch, gave researchers and creative-tool developers access to downloadable checkpoints, local inference and integrations with frameworks including Hugging Face Diffusers and ComfyUI. FLUX.1 Kontext Dev, for example, was released as an open-weight model for research and noncommercial use, with generated outputs permitted for commercial purposes under the applicable license. The company continued that pattern with FLUX.2 Dev in late 2025, a 32-billion-parameter open-weight model combining generation and multi-reference editing. Black Forest Labs called it the strongest open-weight image generation and editing model available at launch and released weights, reference inference code and optimized implementations for consumer Nvidia GPUs. FLUX 3 Dev raises the stakes on that evaluation. Previous Dev releases were image models. This one is described as a multimodal backbone spanning video, audio, image and action prediction — meaning a single license will govern whether a company can locally deploy a model that touches both content production and physical machinery. BFL hasn't yet shared information about its license, the parameter count, quantizations or hardware requirements. The company frames open weights as an enterprise feature rather than a community gesture, arguing they enable secure, low-latency local deployment for applications like robotic control systems and let teams adapt FLUX 3 to their own data, products and workflows. The financial backing behind FLUX 3 is worth noting alongside the technical claims. Black Forest Labs is valued at $3.25 billion and has raised more than $450 million from investors including a16z, AMP, Salesforce Ventures, Nvidia, General Catalyst, Adobe Ventures, Figma Ventures, Canva and Deutsche Telekom's T.Capital.

AINews

Rethinking Legal Education in the AI Era

49 points 13 comments on Hacker News · law.uchicago.edu

Hacker NewsJuly 23, 20261 min read

AINews

Runway launches AI model router as generative media gets crowded

The Media Router is a tool that automatically selects the best image, video, or audio generation model for a request based on whether a developer prioritizes quality, speed or cost.

TechCrunchJuly 23, 20261 min read

AIOpen Source

The AI compute gap: Enterprises are buying infrastructure faster than they can measure what it costs

Across 107 enterprises, AI infrastructure spending is accelerating well ahead of the ability to see or steer its economics. Most organizations run their AI on a familiar base of hyperscalers and model-provider APIs, yet the next dollar is aimed at specialized compute almost none of them use today; a majority intend to switch or add providers within the year, many within a quarter. Buying decisions turn on integration and total cost of ownership rather than headline token price — which is fortunate, because most enterprises cannot yet see their unit economics clearly: GPUs sit at half utilization or less, and fewer than half rigorously track what their compute actually costs. The result is a compute gap — heavy, fast-moving investment running ahead of the visibility needed to control it. This wave of VentureBeat Pulse Research examines enterprise AI infrastructure and compute: where organizations are in their deployment journey, what they run AI on today, how satisfied they are, what would make them switch, where they plan to evaluate their investments, and — most revealingly — how well they can measure and control the economics of the compute underneath it all. The central finding is a compute gap — the distance between how aggressively enterprises are investing in AI infrastructure and how little of its economics they can see. Only about one in five (21%) run AI in production at scale, yet spending intentions are outrunning that maturity: the single largest planned area enterprises plan to evaluate over the next year is AI-specialized clouds (45%), a layer almost none of these enterprises use today. Meanwhile the compute already in place runs cold — 83% report GPU utilization of 50% or less — and fewer than half (44%) can rigorously track what their AI compute costs. Enterprises are buying more infrastructure faster than they can account for what they already own. Enterprises are not settled on their infrastructure vendors, either: A clear majority (64%) plan to switch or add an infrastructure provider within twelve months, and 38% within the next quarter — unusually high churn intent for a category this foundational. When they choose, they choose on integration with the existing stack (41%) and total cost of ownership (35%), not on headline price: cost per million tokens is the deciding factor for just 8%. And the frontier constraint that will shape the next round of decisions — the shift from GPU compute to memory bandwidth as inference scales — is barely on the radar, with roughly one in five enterprises either unaware of it or yet to address it. This report is one of five in VentureBeat Research's Q2 study of the agentic stack. Cost is the control with the least instrumentation: More than eight in ten GPU operators report utilization at half capacity or less, and a minority rigorously track compute cost and return. The executive summary, "VentureBeat Research: Where enterprise AI agent governance hasn't caught up," places that finding in the full pattern. Methodology VentureBeat fielded this survey as part of its ongoing Pulse Research series, this survey focused on enterprise AI infrastructure, compute, and inference economics. Responses are filtered to organizations with more than 100 employees (n=107; the survey’s smallest size band, 1–100 employees, is excluded), drawn from a single Q2 2026 (June) wave. Because this is one wave rather than a pooled multi-month sample, the report reads cross-sectionally and does not infer month-over-month trends. Several questions were multiple-select, so those shares can sum to more than 100%. By organization size the sample concentrates in the mid-market: 101–250 employees (36%) and 251–1,000 (27%) lead, with 1,001–5,000 (22%), 5,001–10,000 (8%), and 10,001+ (7%) above them. By role it spans managers (38%), individual contributors (28%), VPs and directors (19%), and the C-suite (13%); on purchasing authority it is buyer-credible, with 45% final decision-makers and another 30% recommenders or influencers for AI solutions. Technology/Software is the largest industry at 26%, followed by Healthcare/Life Sciences (15%), Financial Services (13%), and Retail/E-commerce (12%). At 107 respondents the sample is large enough to read directionally but should be treated as a directional signal rather than a precise measurement; it is self-selected and is not a probability sample. It also skews toward the mid-market and toward earlier-stage adopters, so it is best read as the view from organizations actively building out AI infrastructure rather than from the largest hyperscale operators. Finding 1: Ambition outpaces production Only one in five run AI in production at scale We asked where organizations sit in their AI deployment journey. Most are still building toward production rather than operating at scale. The maturity curve is front-loaded. Three-quarters of enterprises (76%) are either experimenting or running only some workloads in production, and just 21% describe AI in production at scale. This matters for everything that follows: the infrastructure decisions in this report are being made largely by organizations still early in deployment, whose compute footprint — and whose costs — are about to grow. The evaluation and switching intentions in Findings 3 and 4 are the leading edge of that build-out, not the settled preferences of operators who have already found what works. Finding 2: Enterprises run on hyperscalers and model APIs The specialized GPU clouds barely register — today We asked which providers and platforms enterprises currently use to run their AI. The answer is a familiar one: the incumbents. The current stack is hyperscaler-and-API. Google Cloud leads at 48%, and the general-purpose clouds (Google, Microsoft, AWS, Oracle) together with the major model APIs (Gemini, OpenAI, Anthropic) account for essentially all current deployment. The specialized “neocloud” GPU providers that dominate AI-infrastructure headlines — CoreWeave, Lambda, Crusoe, Nebius and peers — register at or near zero among these enterprises today. Only 6% run their own on-prem GPU clusters and 4% a custom open-source stack. Enterprises are, for now, running AI on the providers they already buy from — which makes the evaluation intentions in Finding 3 all the more striking. (A note on reading these shares. As described in the methodology section, this sample is self-selected and skews mid-market, and this question counted every provider a respondent uses — an average of 2.1 selections each — so the figures measure presence in the stack rather than spending or primary status. A sample built this way will show a different provider mix than a spend-weighted census of the broader market; Google's strength here, for example, is consistent with its long-standing position among smaller enterprises building on AI. Read these shares as a portrait of what this AI-active cohort runs today, and treat gaps between these figures and industry-wide market share estimates as a property of the sample rather than a contradiction of either.) Finding 3: The next dollar goes to infrastructure they don’t yet run AI-specialized clouds top the evaluations list We asked where enterprises planned to evaluate AI infrastructure over the next 12 months. Their answers point away from the stack they run today. Here is the report’s sharpest tension. The single most-cited planned evaluation area — AI-specialized clouds, at 45% — is the very category almost none of these enterprises use today (Finding 2). Nearly a third (32%) intend to evaluate non-Nvidia accelerators, and 28% in next-generation Nvidia silicon; even decentralized compute networks (16%) and sovereign compute (11%) draw meaningful interest. Read against current usage, this is not incremental — it is the leading edge of a re-platforming. The direction-of-travel question tells the same story: every infrastructure approach is net-expanding, but specialized AI clouds carry the highest net momentum (+24), edging out even the hyperscalers (+22). Enterprises are preparing to move a meaningful share of AI compute off the general-purpose cloud. This continues a trend we saw in our April-May survey wave. Back then, usage of the AI-specialized clouds was equally marginal — CoreWeave at 3%, Lambda at 4%, Crusoe at 2% of enterprises. When we asked enterprises what change they planned in their AI infrastructure strategy over the next twelve months, the most-cited answer was moving workloads to specialized AI clouds, at 33%. Asked in April-May which emerging compute option they were most likely to evaluate AI-specialized clouds again drew the most responses. Two waves, two differently worded questions, one consistent picture: the type of cloud enterprises are most eager to assess is the type they have barely begun to use. Finding 4: A switching wave is building Six in 10 plan to change providers within a year — many within a quarter We asked whether and when enterprises plan to switch or add an infrastructure provider. Very few intend to stand still. For a category as foundational as compute, this is a remarkable amount of intended movement. Only 36% have no plans to change, meaning a clear majority (64%) intend to switch or add a provider within twelve months — and 38% within the next quarter alone. Where that interest points is telling: the providers drawing the most switching consideration are again the incumbents — Microsoft Azure and Google Cloud (33% each), OpenAI (30%), and Gemini (22%) — which suggests much of the near-term movement is reshuffling among the majors and consolidating spend rather than defecting to new entrants. The neocloud interest in Finding 3 is a 12-month evaluation thesis; the switching in the next quarter is mostly incumbents trading share. ( Method note: Respondents who selected both "no plans to change" and a specific switching window are counted as switchers, on the logic that naming a timeframe is the more specific answer; three respondents were reclassified under this rule. ) Finding 5: Nobody buys on token price Integration and total cost of ownership decide — not sticker price We asked what matters most when enterprises select an AI infrastructure provider. Headline price finished last. Enterprises do not buy AI infrastructure on pricing, which is the place vendors compete on hardest. Integration with the existing stack (41%) and total cost of ownership (35%) dominate, while the headline metric — cost per million tokens — is the deciding factor for just 8%, dead last. The pattern is coherent: buyers are optimizing for how a provider fits and what it truly costs to operate, not for the advertised unit rate. It also foreshadows Finding 7 — enterprises say TCO matters most, yet most cannot yet measure it rigorously. The stated priority and the measured capability are out of step. Finding 6: Expensive GPUs, idle most of the time 83% report GPU utilization of 50% or less We asked what share of their GPU capacity enterprises actually utilize. The answer is a well-known but rarely quantified inefficiency. Disclosure: Band percentages count every selection against all 107 qualified respondents; 14 respondents selected more than one band, so bands overlap. At the respondent level, 83 of the 100 GPU-operating enterprises reported utilization at or below 50% The compute already in place runs cold. Adding the bands at or below half capacity, 83% of enterprises that operate GPUs report utilization of 50% or less, and nearly half (49%) run at 25% or below. Only 12% clear the 50% mark, and a further 8% do not measure utilization at all. Idle accelerators are expensive accelerators, and this is the clearest single measure of the compute gap: enterprises are planning to buy more GPUs and specialized compute (Finding 3) while the capacity they already own sits substantially unused. The efficiency headroom in the current fleet is large — and largely unmeasured. Finding 7: Spending fast, measuring slowly Fewer than half rigorously track what their compute costs We asked whether enterprises can quantify the cost and return of their AI infrastructure spend, and how satisfied they are with what they run. Confidence in the ledger lags the spending. Measurement trails money. Fewer than half of enterprises (44%) rigorously track the cost and return of their AI compute; the majority track only partially (39%), cannot quantify it yet (20%), or have not prioritized it (6%). That gap is consequential given Finding 5, where total cost of ownership was the second-ranked buying criterion — enterprises are choosing providers on an economic basis they mostly cannot yet measure. Satisfaction with current infrastructure is moderately positive but not enthusiastic: on a five-point scale, overall satisfaction averages 4.0, with ease of implementation (3.8) and value for money (3.9) trailing slightly — the softness landing, tellingly, on cost. Enterprises are spending quickly and accounting slowly. Finding 8: The next bottleneck few are watching As inference shifts from compute to memory, the field scatters Finally, we asked how enterprises would address the emerging constraint in large-scale inference — the shift from GPU compute to memory, specifically KV-cache capacity. The responses reveal a frontier that is not yet a priority. The memory frontier is real but barely governed. Asked which approach they would rely on as the binding constraint in inference shifts from compute to memory bandwidth, enterprises scatter: Dell leads at 31%, Nvidia follows at 16%, and the rest fragments across storage vendors, open-source tooling, and model-level efficiency techniques. Most telling is that roughly one in five (18%) either do not recognize the constraint or have not begun to address it. For a shift that will reshape inference cost and architecture, this is an early and unsettled market — and, consistent with the measurement gap in Finding 7, one where many enterprises simply do not yet have a view. It is the next chapter of the compute gap, arriving before most have closed the current one. The bottom line: A compute gap that faster spending will widen, not close Organizations with more than 100 employees are investing in AI infrastructure faster than they can measure it. Most are still early in deployment, yet their spending intentions point past their current stack — toward specialized clouds and alternative accelerators almost none of them run today — and a clear majority intend to change providers within the year. They buy on integration and total cost of ownership rather than headline price, which is rational; the difficulty is that most cannot yet see those economics clearly. The visibility gap is concrete. The GPUs enterprises already own run at half utilization or less for the overwhelming majority, and fewer than half can rigorously track what their compute costs or returns. Satisfaction is decent but unenthusiastic, softest on value for money — the dimension hardest to judge without measurement. And the next constraint, the shift from compute to memory in large-scale inference, is arriving while most enterprises are still unaware of it. At 107 respondents in a single Q2 wave this is a directional read, skewed toward the mid-market and earlier-stage adopters — but the direction is consistent: the appetite to spend is running well ahead of the instrumentation to spend well. The compute gap is not a capacity problem that more hardware will solve on its own; it is, first, a problem of seeing what the hardware already costs. The open question for later waves is whether enterprises build that visibility before the re-platforming arrives — or buy the next layer of infrastructure as blind to its economics as the last. Based on survey responses from 107 qualified enterprise respondents (100+ employees), drawn from a single Q2 2026 (June) wave. Because this is one wave rather than a pooled multi-month sample, the results read cross-sectionally rather than as a month-over-month trend, and at 107 respondents this is a directional signal rather than a precise measurement — the sample is self-selected, skews mid-market, and leans toward earlier-stage adopters rather than the largest hyperscale operators. Respondents include managers, individual contributors, VPs/directors, and the C-suite, with buyer-credible purchasing authority, across Technology/Software, Healthcare/Life Sciences, Financial Services, Retail/E-commerce, and other industries.

VentureBeatJuly 23, 202613 min read

AIOpen Source

The AI context gap: Enterprise AI organizations have a trust problem, not a retrieval problem — and most are still building the fix

Across 101 enterprises, the infrastructure that feeds AI agents their business context is being built faster than it can be trusted. Retrieval-augmented generation is already the default context source, and provider-native retrieval has quietly overtaken the dedicated vector databases that define the category — yet a majority of enterprises have already watched their agents produce confident, wrong answers traced to missing or inconsistent context. A governed semantic layer is emerging as the fix, but most are still building it; the field is converging on hybrid retrieval; and even as provider-native tools lead in practice, a plurality say they intend to keep best-of-breed. The result is a context gap — agents that sound authoritative running on a foundation their owners do not yet fully trust. This wave of VentureBeat Pulse Research examines the enterprise RAG and context layer: what feeds AI agents their business context, which retrieval systems enterprises run, how they buy and measure them, where the architecture is heading, and — most revealingly — how often that context is already failing them. The central finding is a context gap — the distance between how confidently enterprise agents answer and how reliable the context beneath them actually is. A majority of enterprises (57%) report that in the past six months their AI agents produced confident but wrong answers they traced to missing or inconsistent business context, and more than half of those said it happened more than once. This is not a fringe failure: retrieval is the primary context source for 38% of enterprises, more than any other approach, so when retrieval is thin or inconsistent, the errors it produces are wearing the agent’s authority. The infrastructure to fix it is being built — 58% already run or are building a governed semantic layer — but for most it is not yet in production. Underneath, the market is consolidating in a direction that surprises. Provider-native retrieval — OpenAI’s file search (40%) and Google’s Vertex AI Search (38%) — already leads every dedicated vector database, and enterprises expect hybrid retrieval to dominate by the end of 2026 (34%). Yet a plurality (36%) say they intend to keep best-of-breed standalone tools rather than consolidate onto a provider’s native context stack, and a majority (57%) plan to switch or add a provider within the year. Stated preference and actual usage are pulling in opposite directions — the market is buying provider-native while insisting it wants independence. Methodology VentureBeat fielded this survey as part of its ongoing Pulse Research series. This survey focused on enterprise RAG infrastructure and the context layer — the retrieval systems, semantic layers, and context sources that feed AI agents. Responses are filtered to organizations with more than 100 employees (n=101); the survey drew no responses from organizations of 100 or fewer, so the full sample qualifies. All responses are from a single Q2 2026 (June) wave, so the report reads cross-sectionally and does not infer month-over-month trends. Several questions were multiple-select, so those shares can sum to more than 100%. By organization size the sample concentrates in the mid-market: 251–1,000 employees (31%) and 101–250 (31%) lead, with 1,001–5,000 (20%), 5,001–10,000 (12%), and 10,001+ (7%) above them. By role it spans managers (39%), individual contributors (27%), the C-suite (16%), and VPs and directors (14%); on purchasing authority it is buyer-credible, with 46% final decision-makers and another 26% recommenders or influencers. Technology/Software is the largest industry at 20%, followed by Healthcare/Life Sciences (11%) and a broad spread across retail, transportation, financial services, manufacturing, and education. At 101 respondents this is a modest sample and should be read as a directional signal rather than a precise measurement; it is self-selected and is not a probability sample. It is best read as the view from organizations actively standing up RAG and context infrastructure rather than from the largest operators. Finding 1: Confident and wrong More than half have traced agent errors to bad context We asked whether, in the past six months, enterprises had traced a confident but wrong agent answer to missing or inconsistent business context. Most had. This is the report’s defining number. A majority of enterprises (57%) have already had an AI agent produce a confident, wrong answer they traced to bad context — wrong metrics, stale definitions, or missing documents — and more than half of those have seen it happen more than once. Only 28% report no such failure, and a small remainder either don’t run agents on enterprise data or don’t trace root cause closely enough to know. The failure mode is specific and dangerous: the model is not obviously hallucinating; it is confidently wrong because the context feeding it was thin or inconsistent. Everything else in this report — what enterprises retrieve, how they govern it, and what they plan to build — is downstream of this problem. Finding 2: RAG is the default context source Retrieval feeds more agents than any other method We asked what an enterprise’s AI agents primarily use to understand its data. Retrieval leads by a wide margin. Retrieval is the backbone of enterprise context. For 38% of organizations, RAG over documents or a vector index is the primary way agents understand the business — nearly twice the share of the next approach, a governed semantic layer or ontology (21%). Mixed approaches (14%), direct live-system queries (10%), and long-context loading (6%) fill out the rest, and only 2% let agents run on the model’s general knowledge alone. The concentration matters in light of Finding 1: because so much enterprise context flows through retrieval, the quality of that retrieval is the quality of the answer. When RAG is the default source, thin retrieval is not an edge case — it is the main failure surface. One approach is notable for its absence from these answers: customizing model weights, also known as fine-tuning. Every leading source of business context is injected at run time. Our most recent direct measurement of fine-tuning comes from our April–May survey wave (a separate survey, n=136), where fine-tuning capabilities ranked last of six factors in model selection at 5% — even as 26% of that sample still named fine-tuning and customization an investment they expect to grow. Fine-tuning has fallen out of the primary selection conversation; context injection is how enterprises make agents knowledgeable about their business. Finding 3: Provider-native retrieval already leads the vector databases OpenAI file search and vertex AI search top the dedicated tools We asked which retrieval systems enterprises run in production today. The answer favors the model providers and hyperscalers over the specialists. The dedicated vector database is no longer the center of the RAG stack. OpenAI’s file search (40%) and Google’s Vertex AI Search (38%) lead — provider-native and hyperscaler-native retrieval — ahead of every purpose-built vector database. Among the specialists, the most-used is the one enterprises already run for other reasons (Elasticsearch/OpenSearch, 20%) and the open, embedded option (pgvector, 12%); the pure-play vector databases that define the category — Weaviate, Qdrant, Pinecone, Milvus — each sit in single digits to low double digits. Notably, 13% of enterprises say they still run no production RAG at all. As with the platforms in the parallel infrastructure wave, enterprises are gravitating to retrieval that comes bundled with tools they already buy. The shape of this finding held across both Q2 waves. In April–May (n=161), provider-built retrieval led usage there too, while every dedicated vector database remained marginal — the most-used standalone vector database peaked at 8% of that sample — and the hybrid, pluralistic future was already the consensus expectation (34% expected hybrid retrieval to dominate, with another 29% expecting multiple architectures by use case). Two waves, consistent picture: the category that coined the “vector database” term is being collected by the platforms enterprises already buy from. Finding 4: But they say they want to keep best-of-breed A plurality resist consolidating onto a provider’s native stack We asked how enterprises will respond as model providers bundle retrieval, memory, and orchestration into their platforms. Their stated intent cuts against their current usage. Here is the tension at the heart of the stack. Even as provider-native retrieval leads in practice (Finding 3), a plurality of enterprises (36%) say they intend to keep best-of-breed standalone tools rather than consolidate onto a provider’s native context stack — well ahead of the 21% who plan to consolidate. Another 21% expect a mix, and 9% intend to build and own the layer themselves. The gap between what enterprises run and what they say they want is the strategic question of the category: they are adopting bundled retrieval for convenience while asserting they will preserve independence. Which impulse wins — the pull of the provider bundle or the stated preference for modular control — will shape the retrieval market more than any single tool. Finding 5: Hybrid retrieval is the consensus bet Vector-only retrieval is already seen as insufficient We asked which retrieval architecture enterprises expect to dominate their production RAG systems by the end of 2026. The field is converging — with a large share still unsure. The architecture is settling on hybrid. A third (34%) expect hybrid retrieval — embeddings combined with reranking and access controls — to dominate their production systems by the end of 2026, three times the 11% who expect vector-only retrieval to prevail. That is a notable signal: the pure vector-search approach that launched the category is already viewed as insufficient on its own, superseded by pipelines that add reranking for accuracy and access controls for governance — the very access controls whose absence produces the failures in Finding 1. Tellingly, the second-largest answer is uncertainty: 17% simply don’t know, and another 14% expect to move beyond a dedicated vector layer entirely toward tool-first or long-context retrieval. The consensus is not a single tool but a layered pipeline — and it is not yet fully formed. Finding 6: The governed context layer is being built now Most run or are building a semantic layer — few in production We asked whether enterprises use a governed semantic or context layer to give agents and BI a shared understanding of their data. Most are on the path; fewer have arrived. The fix for the context gap is under construction. Well over half of enterprises (58%) either run a governed semantic layer in production (25%) or are piloting and building one (34%), and a further 17% are actively evaluating — meaning three-quarters are engaged with the idea in some form. But the balance is telling: more are building than have shipped, so for most enterprises the shared, governed definition layer that would prevent the "confident but wrong" failures of Finding 1 is still a work in progress. The semantic layer is the industry’s answer to inconsistent context; this wave catches it mid-construction, ambition well ahead of production. Finding 7: Bought on ingestion and simplicity, watched for correctness Selection favors operability; monitoring favors correctness and security We asked what matters most when enterprises choose a retrieval system, and what they track once it is running. Both answers lean practical. Enterprises choose retrieval systems on operability. Ease of data ingestion (36%), latency and performance (32%), and operational simplicity (29%) lead the selection criteria — ahead of retrieval accuracy and access control (23% each), the two factors most directly tied to the failures in Finding 1. Once systems are running, the emphasis shifts toward trust: the most-tracked metrics are response correctness (42%) and security and access control (38%), ahead of latency (28%), operational stability (27%), and answer relevance (23%). Satisfaction with current systems is moderately positive but not enthusiastic — on a five-point scale, overall satisfaction averages 4.0, with ease of implementation and value for money both near 3.9. Enterprises buy for how easily a system runs and watch it for whether it can be trusted. Finding 8: A retrieval reshuffle is coming A majority plan to change providers — and the vector specialists are gaining interest We asked whether enterprises plan to change or add a retrieval provider, and which they are considering. The consideration set differs from today’s stack. The retrieval stack is not settled. While 43% have no plans to change, a small majority (57%) intend to switch or add a provider within twelve months, and a quarter (26%) within the next quarter. The consideration set is where it gets interesting: provider-native retrieval still leads what enterprises are evaluating (OpenAI 22%, Vertex AI Search 21%), but the open-source vector specialists punch above their current footprint — Qdrant (14%) and Milvus (13%) draw more switching interest than their present usage (10% and 6%) would suggest. Read with Finding 4, the picture is a market in flux: enterprises run provider-native today, are evaluating a broader field, and say they want to keep their options open. The reshuffle ahead will test whether best-of-breed intent survives contact with the convenience of the bundle. The bottom line: A context gap that more retrieval alone won’t close Organizations with more than 100 employees are wiring agents into their business faster than they can guarantee the context those agents run on. Retrieval is the default source of enterprise context, and it increasingly comes from the model providers and hyperscalers rather than the dedicated vector databases — yet a majority of enterprises have already watched agents answer confidently and wrongly because that context was thin or inconsistent. The failure is not exotic; it is the predictable result of pointing authoritative-sounding agents at an unreliable foundation. The industry’s answer — a governed semantic layer, hybrid retrieval with reranking and access controls — is being built but is mostly not yet in production, and enterprises are pulled between the convenience of provider-native bundles and a stated preference for best-of-breed independence. At 101 respondents in a single Q2 wave this is a directional read, skewed toward the mid-market — but the direction is clear: the context layer is the next contested tier of the AI stack, and right now agents are running ahead of it. The context gap is not a retrieval-volume problem that more documents or bigger indexes will solve on their own; it is a problem of governed, consistent, access-aware context. The open question for later waves is whether enterprises finish building that layer before the confident-but-wrong failures move from the lab into decisions that matter. Based on survey responses from 101 qualified enterprise respondents (100+ employees), drawn from a single Q2 2026 (June) wave. At this sample size the results should be read as a directional signal rather than a precise measurement — it's a self-selected sample, not a probability sample, and skews toward the mid-market. Respondents include managers, individual contributors, VPs/directors, and the C-suite, with strong purchasing authority, across technology, healthcare, retail, transportation, financial services, manufacturing, and education.

VentureBeatJuly 23, 202612 min read

AIOpen Source

The agent evaluation gap: Enterprise AI organizations have a reality-alignment problem, not a coverage problem — and most are shipping to production anyway

Across 157 enterprises, organizations are granting AI agents more autonomy while trusting the evaluations meant to gate that autonomy less. Half have already shipped an agent that passed their internal evaluations and then failed a customer in production; only one in twenty fully trusts automated evaluation today; and the most-cited weakness is that evaluations do not align with real-world outcomes. Yet two-thirds already allow, or are actively engineering toward, deploying agent changes to production on automated evaluation alone — with no human in the loop. The result is an evaluation gap — the distance between how much autonomy enterprises are handing their agents and how far they trust the tests that are supposed to catch the failures. This wave of VentureBeat Pulse Research examines how technical leaders measure agent performance: which reliability and evaluation platforms they use, how they select and trust them, what breaks in production, and how far they are willing to let agents run without a human in the loop. The central finding is an evaluation gap — the distance between the autonomy enterprises are granting their agents and the trust they place in the evaluations meant to govern it. Half of organizations (50%) have, in the past year, deployed an agent or LLM feature that passed their internal evaluations and then caused a customer-facing failure, and a quarter have seen it happen more than once. Trust in the tests themselves is thin: only 5% say they fully trust automated evaluation today, and the single most-cited limitation is that evaluations align poorly with real-world outcomes (29%). Enterprises are discovering that a passing eval is not the same as a working agent. What makes the gap consequential is the direction of travel. Two-thirds of organizations (66%) already permit fully automated, zero-human-in-the-loop deployment for low-risk agents (34%) or are actively engineering their pipelines to allow it within twelve months (33%). At the same time, the evaluation stack that would have to earn that trust is fragmented and immature: the most common primary tools are the model providers’ native evals, tied with having no dedicated tooling at all (17% each); and only about a quarter of enterprises run real-time quality checks on live production traffic. The autonomy is arriving faster than the assurance. Methodology VentureBeat fielded this survey as part of its ongoing Pulse Research series, this survey — the Agentic Reliability & Evals tracker — focused on how technical leaders evaluate agent performance and reliability. Responses are filtered to organizations with 100 or more employees (n=157), drawn from a single survey in June 2026; because this is one wave rather than a pooled multi-month sample, the report reads cross-sectionally and does not infer month-over-month trends. Where questions were multiple-select, those shares can sum to more than 100%. By role the sample is senior and buyer-credible: 38% are final decision-makers for AI purchases and another 34% recommenders or influencers. Product and program managers (15%), consultants and advisors (10%), directors of engineering/IT (8%), and CIOs/CTOs/CISOs (8%) lead the named titles, alongside a large “Other” function (37%). By organization size the sample is mid-market-weighted: 100–499 (37%) and 500–2,499 (27%) employees lead, with 2,500–9,999 (20%), 10,000–49,999 (10%), and 50,000+ (6%) above them. Technology/Software is the largest industry at 23%, followed by Retail/Consumer (15%), Healthcare/Life Sciences (12%), and Manufacturing (10%). At 157 respondents the sample is large enough to read directionally but should be treated as a directional signal rather than a precise measurement; it is self-selected and is not a probability sample. It skews toward the mid-market, so it is best read as the view from organizations actively standing up agent evaluation practices rather than from the largest operators. Note: This survey was rebuilt for the June wave from the earlier “LLM observability and evaluations” survey; because the questions and sample differ, no comparisons are made to the April–May data. Finding 1: A passing eval is not a working agent Half have shipped an agent that passed evals, then failed a customer We asked whether, in the past 12 months, organizations had deployed an agent or LLM feature that passed their internal evaluations but then caused a customer-facing failure. Half of those that run evaluations had. This is the report’s defining number. Half of organizations (50%) have shipped an AI feature that cleared their internal evaluations and then failed in front of a customer — an incorrect output, a broken workflow, or a quality incident — and a quarter have seen it happen more than once. Only 36% report no such failure, and the remainder either run no pre-deployment evaluations (8%) or don’t track the root cause closely enough to know (6%). The failure is precise and expensive: the evaluation said the agent was ready, and it was not. Everything that follows — how enterprises trust their evals, what they monitor, and how much autonomy they grant — is shaped by this experience. Finding 2: Almost no one fully trusts automated evaluation The top complaint: Evals don't match real-world outcomes We asked which limitation most reduces trust in automated agent evaluations today. Only a sliver of enterprises had no complaint at all. Trust in automated evaluation is scarce, and specific. Only 5% of organizations say they fully trust automated evaluation as it stands — meaning 95% name a limitation that holds them back. The most common, at 29%, is the one that most directly explains Finding 1: evaluations align poorly with real-world outcomes, passing agents that later fail. Bias or inconsistency (21%) and a lack of explainability (18%) follow — enterprises cannot always tell why an evaluation reached its verdict — and 17% cite data-leakage or privacy concerns in the evaluation process itself. The tests meant to certify agents are not yet trusted to certify them, which is precisely why the autonomy trajectory in Finding 3 is so striking. Finding 3: The autonomy ceiling is rising anyway Two-thirds already allow, or are building toward, zero-human deployment We asked whether organizations would let an autonomous agent deploy a code or system change to production on automated evaluation results alone, with no human-in-the-loop validation. The trajectory runs straight through the trust gap. Here is the paradox at the heart of the report. Even though almost no one fully trusts automated evaluation (Finding 2), two-thirds of organizations (66%) either already allow zero-human-in-the-loop deployment for low-risk agents (34%) or are actively engineering their pipelines to permit it within a year (33%). Only 22% rule it out for the foreseeable future. The direction is unambiguous: enterprises are moving to let evaluations gate production autonomously — removing the human check — at the same moment they say those evaluations don’t reliably match reality. The autonomy ceiling is rising faster than the assurance beneath it, which is the mechanism by which the false-confidence failures of Finding 1 will scale rather than shrink. Notably, the autonomy bet is not just a small company phenomenon. Splitting the sample by company size, larger enterprises are slightly further down the path toward zero human review than smaller companies (70% versus 64%) and slightly more likely to have shipped an evaluation-passing agent that then failed a customer (54% versus 48%). The assumption that large, regulated organizations are holding the human in the loop longest is, in this sample, backwards. To be sure, these are directional figures, since the survey was not a huge sample — 57 respondents from companies with 2,500+ employees and 100 from companies smaller than that. Finding 4: The evaluation stack is fragmented and provider-led Provider-native evals lead — tied with no dedicated tool at all We asked which agent reliability or evaluation platform enterprises primarily use today. The market has no clear leader — and a large share has nothing dedicated. The evaluation layer is early and unconsolidated. Provider-native tooling leads — OpenAI’s native evals and traces (17%) and Anthropic’s Claude Console evals (13%) together outweigh any independent platform — but it is tied at the top by a striking answer: 17% of enterprises use no dedicated agent-evaluation tooling at all, a notable gap for organizations shipping agents to customers. The specialist evaluation vendors — DeepEval (12%), Braintrust (8%), LangSmith, Weave, Promptfoo, Langfuse, Arize — are scattered across single to low double digits, and 11% have built their own. No independent platform has yet become the category standard, which leaves most enterprises evaluating agents with provider-native tools, home-grown scripts, or nothing. Finding 5: Production monitoring rarely watches output quality Only a quarter run real-time quality checks on live traffic Production monitoring for an AI agent can watch two very different things. It can watch whether the system is functioning — is the agent up and responding, did each request complete, how fast, at what cost, with any errors. Or it can watch whether the agent's output is correct — automated checks that evaluate the content of each answer as it goes out: did the agent give the right answer, take the right action, stay within policy. The distinction matters because a confidently wrong answer is invisible to the first kind of monitoring: the request completes, the response is fast, no error is thrown, and every functioning-metric reads healthy. We asked organizations which kind their live production monitoring is built for today. Grouped by what is actually being watched, the split is stark: 51% of organizations monitor only whether the agent is functioning, while 23% monitor whether its answers are right. Counting the ad-hoc reviewers and the don't-knows, roughly three-quarters of organizations run no automated, real-time evaluation of output correctness in production — they can see that the system is up and what it costs, and they are taking the correctness of its answers on faith. That blind spot is the runtime counterpart to the pre-deployment gap in Finding 1: the same organizations engineering the human out of the deployment decision mostly cannot see, in real time, when the deployed agent starts getting things wrong. Finding 6: Bought on cost, measured on consistency Price and integration drive selection; evaluation consistency is the goal We asked what most influenced enterprises’ choice of an evaluation vendor, and what they treat as their primary measure of success. Both answers are pragmatic. Enterprises buy evaluation tooling on economics and trust it on repeatability. Cost of evaluations (28%) narrowly leads selection, just ahead of ease of integration (27%) and evaluation accuracy (24%) — breadth of observability (13%) and vendor roadmap (4%) matter far less. On what success looks like, more than a third (36%) name evaluation consistency — getting the same verdict on the same behavior every time — well ahead of speed of experimentation (19%), reduction in failures (18%), production visibility (13%), and compliance (11%). The emphasis on consistency is telling: before enterprises can trust an evaluation’s verdict, they need it to be stable — the very property whose absence (bias and inconsistency) ranked among the top trust limitations in Finding 2. Satisfaction with current tooling is only moderate, averaging 3.8 on a five-point scale across overall satisfaction, ease of implementation, and value for money. Finding 7: The next dollar goes to humans and observability Investment is flowing to oversight, not just automation We asked which reliability and evaluation investment will grow most over the next year. The money is going toward watching agents more closely — including with people. The second-largest planned investment — behind only production observability — is human review workflows, at 26%. Read against Finding 1, that is the report's quietest contradiction: at the same moment two-thirds of enterprises are engineering the human out of the deployment decision, more of them plan to grow spending on human reviewers (26%) than on the automated evaluation pipelines (16%) that would replace them. The zero-human trajectory and the human-review budget are rising in the same companies at the same time. Indeed, only 8% report that their budget is not increasing. Taken together, enterprises are hedging: building toward autonomy while spending to watch agents more closely and keep humans available for the calls that automated evaluation cannot yet be trusted to make. Finding 8: A tooling reshuffle is coming Nearly two-thirds plan to adopt or switch platforms within a year We asked whether enterprises plan to adopt a new, additional, or replacement evaluation platform, and which they are considering. Few intend to stand pat. The evaluation market is wide open. While 36% have no plans to change, a clear majority (64%) intend to adopt a new, additional, or replacement platform within twelve months, and 31% within the next quarter. The consideration set points where current usage is thinnest: Confident AI’s DeepEval leads what enterprises are evaluating (20%), ahead of OpenAI’s native evals (13%) and Braintrust (9%) — the open-source specialists drawing more interest than their present footprint. Given that so many enterprises today rely on provider-native tools or nothing at all (Finding 4), this is less a defection than a first real wave of tooling adoption — the moment the evaluation layer starts to consolidate. Which platforms earn that trust, in a market where almost no one trusts automated evaluation yet, is the open question this series will keep tracking. The bottom line: An evaluation gap that autonomy will widen, not close Organizations with 100 or more employees are granting AI agents more independence than they trust their evaluations to support. Half have already shipped an agent that passed its evals and then failed a customer; almost none fully trust automated evaluation, chiefly because it doesn’t match real-world outcomes; and most watch production for uptime and cost rather than for whether the agent’s answers are right. Yet two-thirds already allow, or are actively building toward, deploying to production on automated evaluation alone. The vendor market is early and unsettled: the most common primary evaluation tools are provider-native evals, tied with no dedicated tooling at all, and a clear majority plan to adopt or switch platforms within the year. Encouragingly, the next dollar is going to observability and — pointedly — human review, suggesting enterprises sense the gap even as they engineer past it. At 157 respondents in a single wave this is a directional read, skewed toward the mid-market — but the direction is clear: autonomy is being granted on the strength of evaluations that the people granting it do not yet trust. The evaluation gap is not a coverage problem that more tests alone will close; it is a problem of evaluations that reflect reality and can be trusted to gate it. The open question for later waves is whether assurance catches up to autonomy — or whether the false-confidence failures move from customer incidents into changes that deploy themselves. Based on survey responses from 157 qualified enterprise respondents (100+ employees), drawn from a single June 2026 wave. This is a directional read rather than a precise measurement — the sample is self-selected, not a probability sample, and skews toward the mid-market. Respondents include product and program managers, consultants and advisors, directors of engineering/IT, and CIOs/CTOs/CISOs, among other functions, across technology/software, retail/consumer, healthcare/life sciences, manufacturing, and other industries.

VentureBeatJuly 23, 202612 min read

AINews

Meta launched a new AI optimism ad set to a song about human extinction

David Bowie's song "Five Years," which Meta used in a supposedly inspiring advertisement, is about humans learning that they have five years left to live before the apocalypse.

TechCrunchJuly 23, 20261 min read

AINews

Google just had its first negative cash flow quarter due to massive AI spending

Google continues to report big quarterly revenue, but its AI spending has skyrocketed.

Ars TechnicaJuly 23, 20261 min read

AIOpen Source

Show HN: Palmier Pro – Open-source macOS video editor built for AI

Hi HN, we are Marcos and Harrison, cofounders of Palmier ( https://palmier.io ). We are building Palmier Pro, an open source macOS video editor, with built-in AI generation and a local MCP server that connects to your agent. Here are a few demos: - Making some AI transitions: https://www.youtube.com/watch?v=hbM_-eR1GX4 - Multicam editing with Codex: https://www.youtube.com/watch?v=SjS2q2LT1q8 - Cutting long form clips into shorts: https://www.youtube.com/watch?v=PR66eN2ouuQ We built Palmier Pro as an internal tool when we were making AI launch videos for other startups. The main problem it solved in the beginning was the back-and-forth between AI generation platform and video editor. The iteration loop was awkward: AI videos → download → import to editor → edit → realize we need to change the AI video → repeat. So we built a minimal video editor where we could let Claude generate AI videos inside the editor. As we gave more and more tools to the agent, we wanted to push to see what else agents can do in the video editing space. So today, your Claude/Codex can: - Manage projects inside Palmier Pro - Import media from a public URL or filesystem to the project, and organize them in folders - Search media (by embedding footages using SigLIP2 running locally) - Edit the timeline (tracks/clips/keyframes operations) - Generate images, videos, sound effects, captions, music - Export videos There are two ways for LLMs to interact with the editor: by connecting to the local MCP server, or using the in-app chat. Both use the same tools and APIs exposed by the video editor. We have seen people using MCP server to connect to their own workflow to automate massive-scale video editing (e.g. given this same podcast style, replicate it with other footages that I have). We have also seen people using the in-app chat where it lives closer to the editor UI, with lower latency for faster iteration. We don't believe that AI is going to replace human creativity (nor should it), but where it can really help is in automating the gruntwork in video editing. Most work in the editing workflow is more mechanical than creative. Our vision is to build a video editor where AI can enable more individuals to create. Throughout our experiments, AI is not very good at creative editing, but given a pattern (transcription-based, beat-based), it can do a decent job at rough cut. We built Palmier Pro in Swift as an MVP because we wanted better performance and minimal dependencies (no nodejs/webview), with some native macOS APIs like SpeechAnalyzer and CoreML for us to run some models locally. We use SpeechAnalyzer for local transcriber, SigLip2 to embed video frames, beat_this for beat detection, Silero VAD for silence detection, all running locally. The tradeoff is that we don't support Linux or Windows at the moment. Palmier Pro is open source and free to try out (macOS 26 only at the moment, though - we'll support more platforms over time, but for now are focusing on iterating the core product). No login required except for the AI generation features, which route requests to our backend. We are offering free credits on sign up so you can try out the AI generation as well. We'd love to hear your feedback!

Hacker NewsJuly 23, 20263 min read

AINews

AI arms race in line for a reckoning after OpenAI hacking incident

Aggressive training techniques sharpens threat of bad behavior by leading models.

Ars TechnicaJuly 23, 20261 min read

AINews

DARPA, U.S. Air Force fly AI-controlled F-16

213 points 235 comments on Hacker News · darpa.mil

Hacker NewsJuly 23, 20261 min read

AISecurity Advisory

Claude Cowork Flaw Could Let AI Agent Escape Its VM and Access Mac Files

Cybersecurity researchers have uncovered a sandbox escape vulnerability in Anthropic's Claude Cowork that makes it possible to break out of the confines of a Linux virtual machine (VM) within which the agent runs to read or write files anywhere on the Mac. Accomplish AI, which shared details of the vulnerability with The Hacker News ahead of publication, said about 500,000 macOS users running

The Hacker NewsJuly 23, 20261 min read

Artificial Intelligence

Black Forest Labs launches FLUX 3 capable of generating images and 20-second video with audio — but in limited release to start

Rethinking Legal Education in the AI Era

Runway launches AI model router as generative media gets crowded

The AI compute gap: Enterprises are buying infrastructure faster than they can measure what it costs

The AI context gap: Enterprise AI organizations have a trust problem, not a retrieval problem — and most are still building the fix

The agent evaluation gap: Enterprise AI organizations have a reality-alignment problem, not a coverage problem — and most are shipping to production anyway

Meta launched a new AI optimism ad set to a song about human extinction

Google just had its first negative cash flow quarter due to massive AI spending

Show HN: Palmier Pro – Open-source macOS video editor built for AI

AI arms race in line for a reckoning after OpenAI hacking incident

DARPA, U.S. Air Force fly AI-controlled F-16

Claude Cowork Flaw Could Let AI Agent Escape Its VM and Access Mac Files

Top Discussion

Filters