Tootfinder

Spark SQL pipe (|>) for Spark 4.0.0?!
https://issues.apache.org/jira/browse/SPARK-49555
https://

With Apache NiFi, a multimodal data pipelining tool, you can assemble existing and/or custom Java & Python processors into a variety of flows. Join Lester Martin at Berlin Buzzwords this year and watch a rich data pipeline be constructed from Kafka, stored using the Apache Iceberg table format and consumed from Trino.
Learn more:

The inner working of parquette/arrow data in R: #rstats

Unpacking Arrow Datasets – Notes from a data witch
A comment on how Datasets work in Apache Arrow. I’m not really sure who the audience for this one. Am I just talking to myself? Probably.

StreamLink: Large-Language-Model Driven Distributed Data Engineering System
Dawei Feng, Di Mei, Huiri Tan, Lei Ren, Xianying Lou, Zhangxi Tan
https://arxiv.org/abs/2505.21575

StreamLink: Large-Language-Model Driven Distributed Data Engineering System
Large Language Models (LLMs) have shown remarkable proficiency in natural language understanding (NLU), opening doors for innovative applications. We introduce StreamLink - an LLM-driven distributed data system designed to improve the efficiency and accessibility of data engineering tasks. We build StreamLink on top of distributed frameworks such as Apache Spark and Hadoop to handle large data at scale. One of the important design philosophies of StreamLink is to respect user data privacy by ut…

Unveiling the Sagittarius Dwarf Spheroidal Galaxy Core with Gaia DR3
Ellie K. H. Toguchi-Tani, Daniel R. Hey, Thomas de Boer, Peter M. Frinchaboy, Daniel Huber
https://arxiv.org/abs/2507.20212

Unveiling the Sagittarius Dwarf Spheroidal Galaxy Core with Gaia DR3
The Sagittarius dwarf spheroidal galaxy provides us with the unique opportunity to study an ongoing Galactic cannibalistic event between our Milky Way Galaxy and a satellite dwarf galaxy. We present the first major membership star catalog of the Sagittarius dwarf core ($\approx$400,000 sources) and Messier 54 ($\approx$2000 sources) with positions, proper motions, and parallaxes from $Gaia$ DR3, supplemented with metallicities from the Apache Point Observatory Galactic Evolution Experiment. We …

#Apache 2.4.64 is released! It fixes some vulnerabilities, listed here:
https://httpd.apache.org/security/vulnerabilities_24.html

Modern applications require search capabilities that go beyond basic text matching. They must be fast, accurate, personalised and context-aware. At this year's Berlin Buzzwords, Saurabh Singh will demonstrate how OpenSearch’s latest AI/ML enhancements and engine improvements enable organisations to build intelligent, scalable search experiences that meet these evolving needs.
Learn more:

From Search to Insight: Leveraging OpenSearch for Scalable, AI-Driven Search Experiences
Observability, log analytics, GenAI systems, and RAG pipelines must query massive volumes of semantic embeddings to retrieve relevant content instantly. Today’s search systems often fall short in handling high-dimensional vector data and similarity search. OpenSearch 3.0 brings significant architectural improvements to address these challenges. Integrating Apache Lucene 10 and JVM 21, the platform delivers 20% faster queries than its 2.x predecessor and 10× the throughput of 1.x versions. Ne…

Apache Fury (incubating)
#java #python

Well, it doesn't look like much but I just switched my infra from Apache to Caddy.
Sometimes doing some admin work is good for the soul.
(Also a preparation to install an AI scraper poison service)

Join Andrew Musselman and Trevor Grant as they present the latest developments in Mahout's new quantum compute layer, Qumat. They will provide an overview of the project, explain why Qumat was developed, and demonstrate its current capabilities. They will also present a demo of Qumat in action and conclude with calls to action for researchers and engineers who are interested in using and contributing to the project.
Learn more:

Palomar and Apache Point Spectrophotometry of Interstellar Comet 3I/ATLAS
Matthew Belyakov, Christoffer Fremling, Matthew J. Graham, Bryce T. Bolin, Mukremin Kilic, Gracyn Jewett, Carey M. Lisse, Carl Ingebretsen, M. Ryleigh Davis, Ian Wong
https://arxiv.org/abs/2507.11720

Palomar and Apache Point Spectrophotometry of Interstellar Comet 3I/ATLAS
On July 1st 2025 the third interstellar object, 3I/ATLAS or C/2025 N1 (ATLAS), was discovered, with an eccentricity of $e=6.15 \pm 0.01$ and perihelion of $q=1.357\pm0.001$ au. We report our initial visible to near-infrared (420-1000 nm) spectrophotometry of 3I/ATLAS using both the Palomar 200 inch telescope and Apache Point Observatory. We measure 3I/ATLAS to have a red spectral slope of 19 %/100 nm in the 420-700 nm range, and a more neutral 6 %/100 nm slope over 700-1000 nm. We detect no not…

Carrot Cache: High-Performance, SSD-Friendly #Caching #Library for #Java

Join Kevin Liang at this year's Berlin Buzzwords, where he will discuss how Apache Solr/Lucene builds dense vector indexes and talk about how he and his team optimised their dense vector setup, sharing the challenges they faced and the best practices they learned along the way.
Learn more: https…

The people committed to DDoSing the #Apache #SpamAssassin RuleQA server seem to have substantial resources. I’ve blocked a lot of them, but they keep coming, asking about things like the May 7 2017 performance of a single rule in one contributor's stats. Not stuff real people want.
Of course…

#ShamelesslyStolenFromTheOtherSite

Mistral launches its first reasoning models: Magistral Small, on Hugging Face under an Apache 2.0 license, and Magistral Medium, in preview on Mistral's Le Chat (Kyle Wiggers/TechCrunch)
https://techcrunch.com/2025/06/10/mistral-releases-a-pai…

Mistral releases a pair of AI reasoning models | TechCrunch
Like other reasoning models, Magistral works through problems step-by-step for improved consistency and reliability across topics such as math and physics.

Been wanting to build it but I haven't had the time recently: Has anyone integrated iocaine or some similar anti AI scraper tools into apache?

Okee!…?
Sone art schlager-hiphop
RAF Camora x Apache 207 - JUPITER
https://www.youtube.com/watch?v=3RvBj77W9sQ
> JUPITER” hier streamen: http…

What's the preferred easy-to-use benchmarking tool these days for testing full HTTP responses? I know ab (apache bench), but it's also very old so I assume there's a new favorite.
This is for mostly informal tests, so ease of use > capability. Must run on Linux CLI.
#PHP

Big Data-Driven Fraud Detection Using Machine Learning and Real-Time Stream Processing
Chen Liu, Hengyu Tang, Zhixiao Yang, Ke Zhou, Sangwhan Cha
https://arxiv.org/abs/2506.02008 …

Big Data-Driven Fraud Detection Using Machine Learning and Real-Time Stream Processing
In the age of digital finance, detecting fraudulent transactions and money laundering is critical for financial institutions. This paper presents a scalable and efficient solution using Big Data tools and machine learning models. We utilize realtime data streaming platforms like Apache Kafka and Flink, distributed processing frameworks such as Apache Spark, and cloud storage services AWS S3 and RDS. A synthetic dataset representing real-world Anti-Money Laundering (AML) challenges is employed t…

Apache Point Observatory follow-up of ACcelerating Candidate ExopLanet host Stars (APO ACCELS): Ages for 166 Accelerating Stars in the Northern Hemisphere
Anne E. Peck (Department of Astronomy, New Mexico State University), Eric L. Nielsen (Department of Astronomy, New Mexico State University), Robert J. De Rosa (European Southern Observatory), William Thompson (National Research Council, Herzberg Astronomy and Astrophysics), Bruce Macintosh (Department of Astronomy and Astrophysics, U…

Apache Point Observatory follow-up of ACcelerating Candidate ExopLanet host Stars (APO ACCELS): Ages for 166 Accelerating Stars in the Northern Hemisphere
Directly imaged substellar companions with well-constrained ages and masses serve as vital empirical benchmarks for planet formation and evolution models. Potential benchmark companions can be identified from astrometric accelerations of their host stars. We use Gaia DR3 and Hipparcos astrometry to identify 166 northern hemisphere stars with astrometric accelerations consistent with a substellar companion between 0.5'' and 1''. For this accelerating sample we identify young stars using APO/ARCE…

A Galactic Self-Portrait: Density Structure and Integrated Properties of the Milky Way Disk
Julie Imig, Jon A. Holtzman, Gail Zasowski, Jianhui Lian, Nicholas F. Boardman, Alexander Stone-Martinez, J. Ted Mackereth, Moire K. M. Prescott, Rachael L. Beaton, Timothy C. Beers, Dmitry Bizyaev, Michael R. Blanton, Katia Cunha, Jos\'e G. Fern\'andez-Trincado, Catherine E. Fielder, Sten Hasselquist, Christian R. Hayes, Misha Haywood, Henrik J\"onsson, Richard R. Lane, Steven R. M…

A Galactic Self-Portrait: Density Structure and Integrated Properties of the Milky Way Disk
The evolution history of the Milky Way disk is imprinted in the ages, positions, and chemical compositions of individual stars. In this study, we derive the intrinsic density distribution of different stellar populations using the final data release of the Apache Point Observatory Galactic Evolution Experiment (APOGEE) survey. A total of 203,197 red giant branch stars are used to sort the stellar disk ($R \leq 20$ kpc) into sub-populations of metallicity ($Δ$[M/H]$= 0.1$ dex), age ($Δ\log(\fr…

As data speeds increase, it has become crucial to detect problems as they happen. At this year's Berlin Buzzwords, Olena Kutsenko explained how to build a real-time anomaly detection system using Apache Kafka for streaming, Apache Flink for processing, and AI for pattern recognition, covering Apache Iceberg for storing historical data to improve models.
Watch the full session:

This https://arxiv.org/abs/2504.06151 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_csOS_…

Zerrow: True Zero-Copy Arrow Pipelines in Bauplan
Bauplan is a FaaS-based lakehouse specifically built for data pipelines: its execution engine uses Apache Arrow for data passing between the nodes in the DAG. While Arrow is known as the "zero copy format", in practice, limited Linux kernel support for shared memory makes it difficult to avoid copying entirely. In this work, we introduce several new techniques to eliminate nearly all copying from pipelines: in particular, we implement a new kernel module that performs de-anonymization, thus eli…

Microservices and Real-Time Processing in Retail IT: A Review of Open-Source Toolchains and Deployment Strategies
Aaditaa Vashisht (Department of Information Science,Engineering, RV College of Engineering, India), Rekha B S (Department of Information Science,Engineering, RV College of Engineering, India)
https://arxiv.org/abs/25…

Microservices and Real-Time Processing in Retail IT: A Review of Open-Source Toolchains and Deployment Strategies
With the rapid pace of digital transformation, the retail industry is increasingly depending on real-time, scalable, and resilient systems to manage financial transactions, analyze customer behavior, and streamline order processing. This literature review explores how modern event-driven and microservices-based architectures, particularly those leveraging Apache Kafka, Spring Boot, MongoDB, and Kubernetes are transforming retail and financial systems. By systematically reviewing academic public…

Heute vor 69 Jahren: Am 8. Juli 1956 kam es zum #Atomtest "Apache". Die Operation #Redwing war eine US-Serie von 17 Atomtestdetonationen von Mai bis Juli 1956. Sie wurden von der Joint Task Force 7 (JTF7) auf den Atollen

Apache Point rapid response characterization of primitive pre-impact detection asteroid 2024 RW$_1$
Carl Ingebretsen, Bryce T. Bolin, Robert Jedicke, Peter Vere\v{s}, Christine H. Chen, Carey M. Lisse, Russet McMillan, Torrie Sutherland, Amanda J. Townsend
https://arxiv.org/abs/2505.23736

Apache Point rapid response characterization of primitive pre-impact detection asteroid 2024 RW$_1$
Pre-impact detection asteroids (PIDAs) may be detected only a few hours before their impact with Earth, providing a brief opportunity to characterize them before impact. We describe the characterization of PIDA 2024 RW$_1$, which was discovered by the Catalina Sky Survey on 2024 September 4 at 05:43 UTC, before it entered the atmosphere near the northern Philippines at 16:39 UTC. We observed 2024 RW$_1$ with the Astrophysical Research Consortium Telescope Imaging Camera on the Apache Point Astr…

Plug. Play. Persist. Inside a Ready-to-Go Havoc C2 Infrastructure
Alessio Di Santo
https://arxiv.org/abs/2507.00189 https://arxiv.org…

Plug. Play. Persist. Inside a Ready-to-Go Havoc C2 Infrastructure
This analysis focuses on a single Azure-hosted Virtual Machine at 52.230.23.114 that the adversary converted into an all-in-one delivery, staging and Command-and-Control node. The host advertises an out-of-date Apache 2.4.52 instance whose open directory exposes phishing lures, PowerShell loaders, Reflective Shell-Code, compiled Havoc Demon implants and a toolbox of lateral-movement binaries; the same server also answers on 8443/80 for encrypted beacon traffic. The web tier is riddled with publ…

At Berlin Buzzwords 2025, Ved Prakash discussed how Siphon transformed their data pipeline using Apache Iceberg to successfully stream quality data into both Snowflake and Clickhouse simultaneously. In this short talk, you’ll learn about their battle-tested architecture, the performance improvements they’ve achieved, and their strategies for maintaining data consistency across two analytics engines.
Watch the full session:

Got slammed by an unidentified but certainly "#AI"-related #distributed #crawler this week, it drove one site's traffic to 10× average. Today I tired of playing Whac-a-Mole and blocked the two bigge…

CityPulse: Real-Time Traffic Data Analytics and Congestion Prediction
Idriss Djiofack Teledjieu, Irzum Shafique
https://arxiv.org/abs/2506.01971 https://…

CityPulse: Real-Time Traffic Data Analytics and Congestion Prediction
CityPulse is a proof-of-concept big data pipeline designed to enable real-time urban mobility analytics using scalable, containerized components -- without reliance on physical sensor infrastructure. The system simulates the ingestion of 11 million traffic-related records representing urban phenomena such as vehicle congestion, GPS coordinates, and weather conditions. Data is ingested through a Dockerized Apache Kafka cluster, coordinated by ZooKeeper, and processed in real time using Apache Sp…

Abundances of P, S, and K in 58 bulge spheroid stars from APOGEE
B. Barbuy, H. Ernandes, A. C. S. Fria\c{c}a, M. S. Camargo, P. da Silva, S. O. Souza, T. Masseron, M. Brauner, D. A. Garcia-Hernandez, J. G. Fernandez-Trincado, K. Cunha, V. V. Smith, A. Peerez-Villegas, C. Chiappini, A. B. A. Queiroz, B. X. Santiago, T. C. Beers, F. Anders, R. P. Schiavon, M. Valentini, D. Minniti, D. Geisler, D. Souto, V. M. Placco, M. Zoccali, S. Feltzing, M. Schultheis, C. Nitschelm

Abundances of P, S, and K in 58 bulge spheroid stars from APOGEE
We have previously studied several elements in 58 selected bulge spheroid stars, based on spectral lines in the H-band. We now derive the abundances of the less-studied elements phosphorus (P; Z=15), sulphur (S; Z=16), and potassium (K; Z=19). The abundances of P, S, and K in 58 bulge spheroid stars are compared both with the results of a previous analysis of the data from the Apache Point Observatory Galactic Evolution Experiment (APOGEE), and with a few available studies of these elements. We…

🇺🇦 #NowPlaying on #BBC6Music's #JamzSupernova
Steven Julien:
🎵 Apache
#StevenJulien
https://stevenjulien.bandcamp.com/track/apache
https://open.spotify.com/track/7dkPQYZ3giMtlZm1K90CYg

Apache Flink is uniquely positioned to serve as the backbone for AI agents, equipping them with the powerful new tool of stream processing. Join Steffen Hoellinger at this year's Berlin Buzzwords to explore how Flink jobs can be transformed into “Agents”—autonomous, goal-driven entities that dynamically interact with data streams, trigger actions, and adapt their behaviour based on real-time insights.
Learn more:

We're thrilled to announce that @… has rejoined Berlin Buzzwords as a Platinum Partner!
Learn more about OpenSearch: https://opensearch.org/

🇺🇦 #NowPlaying on BBCRadio3's #ThisClassicalLife
David Raksin & Johnny Mercer:
🎵 Love Song From "Apache"
#DavidRaksin #JohnnyMercer

OWLS I: The Olin Wilson Legacy Survey
Brett M. Morris, Leslie Hebb, Suzanne L. Hawley, Kathryn Jones, Jake Romney
https://arxiv.org/abs/2507.07330 https://…

OWLS I: The Olin Wilson Legacy Survey
We present initial results from a planned 10 year survey of Ca II H & K emission, using observations made with the ARC 3.5m Telescope at Apache Point Observatory. The primary goal of the survey is to investigate activity cycles in low mass stars. The sample includes stars chosen from the legacy Mount Wilson survey carried out by Olin Wilson more than 50 years ago, together with newly identified planet-host stars and a select sample of early-mid M dwarfs. This paper presents the first four years…

Analysis of Server Throughput For Managed Big Data Analytics Frameworks
Emmanouil Anagnostakis, Polyvios Pratikakis
https://arxiv.org/abs/2506.03854 https:…

Analysis of Server Throughput For Managed Big Data Analytics Frameworks
Managed big data frameworks, such as Apache Spark and Giraph demand a large amount of memory per core to process massive volume datasets effectively. The memory pressure that arises from the big data processing leads to high garbage collection (GC) overhead. Big data analytics frameworks attempt to remove this overhead by offloading objects to storage devices. At the same time, infrastructure providers, trying to address the same problem, attribute more memory to increase memory per instance le…

At this year's Berlin Buzzwords, Michal Gancarski led a workshop demonstrating practical ways to deploy, configure, interact with, and utilise the advanced features of Apache Iceberg.
 
Watch the full session: https://youtu.be/v15EiNQt9R0?si=QDuAZ4NqqmlaUSUK

Join Adrien Grand and Luca Cavanna at this year's Berlin Buzzwords as they share the fascinating journey to the release of version 10.0 of the popular Java search engine Apache Lucene, discussing the ups and downs, the team effort it took to get there, and much more.
Learn more: https://

Join Adrien Grand and Luca Cavanna at this year's Berlin Buzzwords as they share the fascinating journey to the release of version 10.0 of the popular Java search engine Apache Lucene, discussing the ups and downs, the team effort it took to get there, and much more.
Learn more: https://

Apache Solr 9.8 introduces the LLM module, opening the doors to end-to-end natural language query support through vector-backed semantic search (K Nearest Neighbors). At Berlin Buzzwords 2025, Alessandro Benedetti discussed the open-source contributions from both an indexing and query perspective, as well as what's next for Solr in terms of Large Language Model integration.
Watch the full session here:

At Berlin Buzzwords 2025, Javier Ramirez shared the journey of developing QuestDB, an Apache 2.0-licensed open-source time-series database, into a much faster analytical database.
Watch the full session: https://youtu.be/SuxHP3_KOgQ?si=mGVdSKX5tHV81If7

At this year's Berlin Buzzwords, Ilaria Petreti, Anna Ruggero, and Edward Lambe presented an AI Filter Assistant for Statistical Data (SDMX). They demonstrated how large language models can suggest the most effective filters for your natural language queries and assist in refining your results in Apache Solr.
You can watch the full session here: …