# Recommended videos

> **Source**: [Coherent](https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ)

## Browse Collection

1. [What is Coherent Spark?](https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/H5WYl7ZaNvgXCTrR0PTkJg.md) (Video)
2. [Crafting front end using Coherent Validation API](https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/1DMjKEJmLUV6qPqTdFKIsQ.md) (Video)
3. [Traffic Director: Rating and Rules](https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/I0g3sx3MrYnr3VdJIpQ2XA.md) (Video)
4. [Employee Benefits - Spark Shell ](https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/wwkXuHO5AjyjQ9fTIawTyw.md) (Video)
5. [Tech 205: Integrating with LLM](https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/FeEnXeDvUcHvqoXP5ITiCQ.md) (Video)
6. [Batch API using Python SDK (Clone)](https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/8AYIt0ikqUJ4dhFayUnlfg.md) (Video)

---

# VIDEO: Batch API using Python SDK (Clone)

## METADATA

- id: 8AYIt0ikqUJ4dhFayUnlfg
- type: video
- source: Coherent
- content_url: https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/8AYIt0ikqUJ4dhFayUnlfg

- collection_url: https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ

## TRANSCRIPT

- time: 00:00
  url: https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/8AYIt0ikqUJ4dhFayUnlfg?start_time=0
  text: Hi, everybody. My name is Ralph, and in this video,

- time: 00:04
  url: https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/8AYIt0ikqUJ4dhFayUnlfg?start_time=4
  text: I'm going to show you how to run

- time: 00:05
  url: https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/8AYIt0ikqUJ4dhFayUnlfg?start_time=5
  text: batch APIs using the Coherent Python SDK.

- time: 00:08
  url: https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/8AYIt0ikqUJ4dhFayUnlfg?start_time=8
  text: If you're not familiar with batch APIs,

- time: 00:11
  url: https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/8AYIt0ikqUJ4dhFayUnlfg?start_time=11
  text: they are a series of endpoints that allow you to

- time: 00:14
  url: https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/8AYIt0ikqUJ4dhFayUnlfg?start_time=14
  text: collectively process a massive amount of

- time: 00:16
  url: https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/8AYIt0ikqUJ4dhFayUnlfg?start_time=16
  text: data while using a dedicated infrastructure.

- time: 00:20
  url: https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/8AYIt0ikqUJ4dhFayUnlfg?start_time=20
  text: While you could manually orchestrate

- time: 00:22
  url: https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/8AYIt0ikqUJ4dhFayUnlfg?start_time=22
  text: this operation using the API documentation,

- time: 00:25
  url: https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/8AYIt0ikqUJ4dhFayUnlfg?start_time=25
  text: this demo will demonstrate how to achieve

- time: 00:27
  url: https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/8AYIt0ikqUJ4dhFayUnlfg?start_time=27
  text: the exact same results using the Python SDK.

- time: 00:32
  url: https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/8AYIt0ikqUJ4dhFayUnlfg?start_time=32
  text: The Python SDK package can be found in

- time: 00:35
  url: https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/8AYIt0ikqUJ4dhFayUnlfg?start_time=35
  text: the PyPI library under the name

- time: 00:36
  url: https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/8AYIt0ikqUJ4dhFayUnlfg?start_time=36
  text: cspark, short for Coheret Spark.

- time: 00:41
  url: https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/8AYIt0ikqUJ4dhFayUnlfg?start_time=41
  text: As you may probably guess,

- time: 00:43
  url: https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/8AYIt0ikqUJ4dhFayUnlfg?start_time=43
  text: this Python package covers a series of Spark endpoints,

- time: 00:46
  url: https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/8AYIt0ikqUJ4dhFayUnlfg?start_time=46
  text: especially those for batch APIs.

- time: 00:48
  url: https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/8AYIt0ikqUJ4dhFayUnlfg?start_time=48
  text: For more information, you can

- time: 00:50
  url: https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/8AYIt0ikqUJ4dhFayUnlfg?start_time=50
  text: navigate to the main documentation.

- time: 00:53
  url: https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/8AYIt0ikqUJ4dhFayUnlfg?start_time=53
  text: However, for today's demo,

- time: 00:54
  url: https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/8AYIt0ikqUJ4dhFayUnlfg?start_time=54
  text: I will jump right in and show you how it works.

- time: 00:58
  url: https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/8AYIt0ikqUJ4dhFayUnlfg?start_time=58
  text: We are going to be using this Term Life Quote model

- time: 01:02
  url: https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/8AYIt0ikqUJ4dhFayUnlfg?start_time=62
  text: as a Spark service to show you the batch API in actions.

- time: 01:06
  url: https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/8AYIt0ikqUJ4dhFayUnlfg?start_time=66
  text: If we took a quick look here at

- time: 01:08
  url: https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/8AYIt0ikqUJ4dhFayUnlfg?start_time=68
  text: the service to see what happens,

- time: 01:10
  url: https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/8AYIt0ikqUJ4dhFayUnlfg?start_time=70
  text: on the left, you could see a series of inputs.

- time: 01:13
  url: https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/8AYIt0ikqUJ4dhFayUnlfg?start_time=73
  text: Those are the fields we're going to be using.

- time: 01:15
  url: https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/8AYIt0ikqUJ4dhFayUnlfg?start_time=75
  text: If I execute this,

- time: 01:16
  url: https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/8AYIt0ikqUJ4dhFayUnlfg?start_time=76
  text: on the right, you could see the expected outputs.

- time: 01:24
  url: https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/8AYIt0ikqUJ4dhFayUnlfg?start_time=84
  text: Here in VS Code, you can see

- time: 01:26
  url: https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/8AYIt0ikqUJ4dhFayUnlfg?start_time=86
  text: the code base that I'm going to use to run this process.

- time: 01:29
  url: https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/8AYIt0ikqUJ4dhFayUnlfg?start_time=89
  text: Let me walk you through it real quick.

- time: 01:31
  url: https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/8AYIt0ikqUJ4dhFayUnlfg?start_time=91
  text: The .demo folder contains

- time: 01:32
  url: https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/8AYIt0ikqUJ4dhFayUnlfg?start_time=92
  text: the Python environment that we're going to be using.

- time: 01:35
  url: https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/8AYIt0ikqUJ4dhFayUnlfg?start_time=95
  text: It has Python 3.11 and all the dependencies.

- time: 01:39
  url: https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/8AYIt0ikqUJ4dhFayUnlfg?start_time=99
  text: The second item here is the .env file that I'm going to

- time: 01:42
  url: https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/8AYIt0ikqUJ4dhFayUnlfg?start_time=102
  text: use to keep the Spark credentials,

- time: 01:44
  url: https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/8AYIt0ikqUJ4dhFayUnlfg?start_time=104
  text: such as bearer token and base URL.

- time: 01:47
  url: https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/8AYIt0ikqUJ4dhFayUnlfg?start_time=107
  text: Let me show you real quick.

- time: 01:49
  url: https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/8AYIt0ikqUJ4dhFayUnlfg?start_time=109
  text: The inputs.json file here is the dataset we'll be using.

- time: 01:54
  url: https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/8AYIt0ikqUJ4dhFayUnlfg?start_time=114
  text: This dataset contains about 1,000 data points,

- time: 01:57
  url: https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/8AYIt0ikqUJ4dhFayUnlfg?start_time=117
  text: and you could see it's in a tabular format,

- time: 02:01
  url: https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/8AYIt0ikqUJ4dhFayUnlfg?start_time=121
  text: which means you see the input headers first,

- time: 02:04
  url: https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/8AYIt0ikqUJ4dhFayUnlfg?start_time=124
  text: followed by the input values.

- time: 02:07
  url: https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/8AYIt0ikqUJ4dhFayUnlfg?start_time=127
  text: The source code here is the main.py

- time: 02:09
  url: https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/8AYIt0ikqUJ4dhFayUnlfg?start_time=129
  text: that contains the logic to run batch API.

- time: 02:12
  url: https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/8AYIt0ikqUJ4dhFayUnlfg?start_time=132
  text: The requirements here includes

- time: 02:15
  url: https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/8AYIt0ikqUJ4dhFayUnlfg?start_time=135
  text: the additional data dependencies required to run this script.

- time: 02:19
  url: https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/8AYIt0ikqUJ4dhFayUnlfg?start_time=139
  text: Obviously, cspark is one of them.

- time: 02:23
  url: https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/8AYIt0ikqUJ4dhFayUnlfg?start_time=143
  text: Let's open the terminal and see how this is going to work.

- time: 02:27
  url: https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/8AYIt0ikqUJ4dhFayUnlfg?start_time=147
  text: Here, I have my terminal open.

- time: 02:30
  url: https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/8AYIt0ikqUJ4dhFayUnlfg?start_time=150
  text: I want to run this by executing the main.py file.

- time: 02:34
  url: https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/8AYIt0ikqUJ4dhFayUnlfg?start_time=154
  text: Once I do this, it's going to start a process.

- time: 02:37
  url: https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/8AYIt0ikqUJ4dhFayUnlfg?start_time=157
  text: Let me take this full screen so you can appreciate it.

- time: 02:42
  url: https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/8AYIt0ikqUJ4dhFayUnlfg?start_time=162
  text: Like we talked about,

- time: 02:43
  url: https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/8AYIt0ikqUJ4dhFayUnlfg?start_time=163
  text: we created a batch pipeline and pushed 1,000 records to it,

- time: 02:47
  url: https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/8AYIt0ikqUJ4dhFayUnlfg?start_time=167
  text: and we'll wait for it to be complete.

- time: 02:49
  url: https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/8AYIt0ikqUJ4dhFayUnlfg?start_time=169
  text: You can see this was executed super fast and in one go.

- time: 02:54
  url: https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/8AYIt0ikqUJ4dhFayUnlfg?start_time=174
  text: As discussed, this represents a series of endpoints

- time: 02:57
  url: https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/8AYIt0ikqUJ4dhFayUnlfg?start_time=177
  text: that we call collectively to be able to achieve this.

- time: 03:00
  url: https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/8AYIt0ikqUJ4dhFayUnlfg?start_time=180
  text: Let's take a look at the outputs to see what happens.

- time: 03:05
  url: https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/8AYIt0ikqUJ4dhFayUnlfg?start_time=185
  text: Let's minimize this and let's open the outputs.json.

- time: 03:09
  url: https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/8AYIt0ikqUJ4dhFayUnlfg?start_time=189
  text: In this file, you can see it's divided by chunks.

- time: 03:12
  url: https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/8AYIt0ikqUJ4dhFayUnlfg?start_time=192
  text: Each chunk has the inputs field and also the outputs field.

- time: 03:18
  url: https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/8AYIt0ikqUJ4dhFayUnlfg?start_time=198
  text: The inputs field represents the chunks that we submitted as input data,

- time: 03:22
  url: https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/8AYIt0ikqUJ4dhFayUnlfg?start_time=202
  text: and the outputs field is the results we get back from Spark,

- time: 03:26
  url: https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/8AYIt0ikqUJ4dhFayUnlfg?start_time=206
  text: which also came in in a tabular format.

- time: 03:30
  url: https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/8AYIt0ikqUJ4dhFayUnlfg?start_time=210
  text: These output headers are the same as we saw in the API tester.

- time: 03:35
  url: https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/8AYIt0ikqUJ4dhFayUnlfg?start_time=215
  text: We choose to save this as a file,

- time: 03:37
  url: https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/8AYIt0ikqUJ4dhFayUnlfg?start_time=217
  text: but you're free to choose a different persistence layer for your system.

- time: 03:41
  url: https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/8AYIt0ikqUJ4dhFayUnlfg?start_time=221
  text: This was all for the batch API demo.

- time: 03:44
  url: https://coherent.portal.trainn.co/oGECFYpUljYdG0l0CZ6dMQ/8AYIt0ikqUJ4dhFayUnlfg?start_time=224
  text: I hope you had fun and I hope to see you in the next video.

## SUMMARY

# How to Run Batch APIs Using the Coherent Python SDK

Hi, everybody. My name is Ralph, and in this video, I'm going to show you how to run batch APIs using the Coherent Python SDK. If you're not familiar with batch APIs, they are a series of endpoints that allow you to collectively process a massive amount of data using dedicated infrastructure. While you could manually orchestrate this operation using the API documentation, this demo will demonstrate how to achieve the same results using the Python SDK.

## Getting Started with the Python SDK

The Python SDK package can be found in the PyPI library under the name **cspark** , short for Coherent Spark. This package covers a series of endpoints, especially those for batch APIs. For more information, you can navigate to the main documentation. However, for today's demo, we will jump right in and show you how it works.

## Term Life Quote Model as a Spark Service

We will use the Term Life Quote model as a Spark service to demonstrate the batch API in action. On the left, you will see a series of inputs, which are the fields we will be using. Upon execution, the expected outputs will appear on the right.

## Setting Up the Environment

- **Python Environment:** We will use Python 3.11 along with all necessary dependencies.

- **.env File:** This file will store the Spark credentials, such as bearer token and base URL.

- **input.json:** This dataset contains about 1000 data points in a tabular format, with headers followed by values.

- **main.py:** This source code contains the logic to run the batch API.

- **Requirements:** This specifies all dependencies needed to run the process, including cspark.

## Running the Batch API

Let's open the terminal to see how this works. We will run the process using the command:

    python main.py

Once executed, the process will start. We pushed 1000 records to the pipeline, and the execution is super fast and efficient. The batch API calls a series of endpoints collectively to achieve this.

## Viewing the Outputs

Let's take a look at the outputs by opening the **output.json** file. This file is divided into chunks, with each chunk containing input fields and output fields. The input fields are the chunks submitted for processing, and the output fields are the results obtained. The data is in a tabular format, similar to what we saw in the API tester. We chose to persist this as a file, but you are free to choose a different persistence layer for your system.

## Conclusion

This concludes the batch API demo. I hope you found it informative and enjoyable. See you in the next video!