Overview
Stellar network data ingestion in Hubble is done through stellar-etl. stellar-etl reads and transforms Stellar network data into OLAP friendly JSON files.
| Use Case | Recommended Solution | Reason |
|---|---|---|
| Need commonly required data fields and do not want to take burden of setting up entire pipeline | Use output from SDF's stellar-etl instance i.e. Connect to Bigquery and access hubble | It is worth noting that most users will not need to standup and run their own stellar-etl instance. The Stellar Development Foundation provides public access to fully transformed Stellar network data through the public datasets and tables in GCP BigQuery. |
| Need custom data but do not want to take burden of setting up entire pipeline | Use JS-UDF | Stellar-ETL does not parse every single field available in raw XDR, but it does save the raw transaction meta. JS-UDF helps to extract a field directly from XDR |
| Need custom and operationally independent data | Run stellar-etl | Running stellar-etl within your own infrastructure provides a number of benefits. You can: - Have full operational control without dependency on the Stellar Development Foundation for network data - Run modified ETL/ELT pipelines that fit your individual business needs |