Send HTTP Requests with Bash Scripting

Introduction

Bash scripting offers a powerful way to automate tasks and interact with various services, including sending HTTP requests. This guide will equip you with the knowledge and tools to effectively create and execute HTTP requests within your Bash scripts, covering essential concepts and best practices.

Prerequisites

This guide assumes familiarity with basic Bash scripting concepts like variables, operators, loops, and conditional statements. Additionally, understanding of HTTP protocols and request types (GET, POST, PUT, DELETE) is advantageous.

Sending HTTP Requests with Bash

Using curl

The curl command is a widely used tool for sending HTTP requests and transferring data. It offers a comprehensive set of features for constructing various request types and customizing headers, body content, and other parameters.

Example GET request:

curl "https://jsonplaceholder.typicode.com/todos/1"

This command fetches the content of the specified URL and displays it on the console. You can further customize the request by adding options like:

• -X: Specify the request method (e.g., POST, PUT, DELETE)
• -d: Send data with the request body
• -H: Set custom headers
• -o: Save the response to a file

Example POST request with JSON data:

curl -X POST "https://jsonplaceholder.typicode.com/todos/1" \
-H "Content-Type: application/json" \
-d '{"name": "John Doe", "age": 30}'

This code sends a POST request to the specified API endpoint with a JSON payload containing name and age information.

Example downloading a file:

curl -L "https://example.com/download/file.zip" -o file.zip

This command retrieves the file from the URL and saves it as file.zip in the current directory.

Using wget

Another popular option is wget, primarily used for downloading files from web servers. However, it can also be utilized for sending basic HTTP requests and retrieving content.

Example GET request with wget:

wget "https://jsonplaceholder.typicode.com/todos"

This command downloads the content from the specified URL and saves it as index.html by default. You can use the -O option to specify a custom filename.

Using nc (Netcat)

Netcat (nc) is a versatile networking tool that can be used to send raw HTTP requests. Here’s how you can use it:

echo -e "GET /todos/1 HTTP/1.1\nHost: jsonplaceholder.typicode.com\n\n" | nc jsonplaceholder.typicode.com 80

This method opens a TCP connection to the target server on port 80 and sends the raw HTTP request. The response is displayed in the terminal.

Using telnet

telnet is another command-line tool that can be used to send HTTP requests, similar to nc.

#!/bin/bash

# HTTP GET request using telnet
telnet jsonplaceholder.typicode.com 80 <<EOF
GET / HTTP/1.1
Host: jsonplaceholder.typicode.com

EOF

This works similarly to nc, but using the telnet command instead. Note that telnet might not be installed by default on all systems.

Using openssl

openssl is primarily known for its cryptography-related tasks, but it can also be used to send HTTP requests over HTTPS.

openssl s_client -connect jsonplaceholder.typicode.com:443 < /dev/null | telnet

This command initiates an SSL/TLS connection to the server and sends the HTTP request. It’s particularly useful for sending requests to HTTPS endpoints.

Using socat

socat is a multipurpose relay tool that can be used to send HTTP requests as well.

echo -e "GET / HTTP/1.1\nHost: jsonplaceholder.typicode.com\n\n" | socat - TCP:jsonplaceholder.typicode.com:80

socat can relay data between two endpoints. Here, it sends the HTTP request to the specified server and port.

Using lynx

Lynx is a text-based web browser. Although not for everyday web browsing, it can be used to make simple HTTP requests. Here's how to retrieve the data:

lynx -dump https://jsonplaceholder.typicode.com/todos/1

This will display the raw JSON data in the terminal.

Using httpie (if installed)

Httpie is a user-friendly HTTP client. If you have it installed, you can use:

http GET https://jsonplaceholder.typicode.com/todos/1

Sending a POST request to the specified URL with the data name="John Doe" in the request body:

For example, include a Content-Type header with a value of application/json, you can use the following command:

http POST https://jsonplaceholder.typicode.com/todos title="John Doe" 'Content-Type:application/json'

Replace with the actual bearer token you want to use.

If you have a JSON file with the data you want to send, you can use the @ symbol to specify the file path:

http POST https://jsonplaceholder.typicode.com/todos @data.json

This will send the contents of the data.json file as the request body.

Best practices

• Use the appropriate request method for your purpose.
• Set appropriate headers and content types.
• Validate user input before sending requests.
• Handle errors gracefully.
• Use tools like curl and wget where possible.
• Document your scripts clearly.

Conclusion

By mastering the techniques in this guide, you'll be able to leverage Bash scripting for sending HTTP requests, interacting with web services, and automating various online tasks effectively. Remember to choose the appropriate tool and approach based on your specific requirements and experiment with different methods to optimize your scripting workflow.