I have an API in a container and when I create a cluster the api works fine, but the pods constantly restart and there isn't any specific reason why in the logs.
I'm on an m5.Large EC2 instance and using k3d. I was following a demo so after installing k3d I ran this command
k3d cluster create test -p "80:80@loadbalancer"
Here is my deployment file
apiVersion: apps/v1
kind: Deployment
metadata:
name: stock-api
labels:
app: stock-api
spec:
replicas: 1
selector:
matchLabels:
app: stock-api
template:
metadata:
labels:
app: stock-api
spec:
containers:
- name: stock-api
image: mpriv32/stock-api:latest
envFrom:
- secretRef:
name: api-credentials
Service file
apiVersion: v1
kind: Service
metadata:
name: stock-api
labels:
app: stock-api
spec:
type: ClusterIP
ports:
- port: 80
protocol: TCP
targetPort: 80
selector:
app: stock-api
My third file is just my secrets file which I just removed the values for this post
apiVersion: v1
kind: Secret
metadata:
name: api-credentials
stringData:
ACCESS_KEY:
SECRET_KEY:
EMAIL_PASSWORD:
EMAIL_USER:
API_KEY:
I applied all of the files and the API works fine, but then my pods constantly restart.
First I ran this command and the reason I got stock information is because I had a print statement in my application to test the api response and forgot to remove it.
kubectl logs -p stock-api-7f5c45776b-gc67c
{'Company': {'S': 'AAPL'}, 'DailyPrice': {'S': '166.02'}}
Getting the logs didn't help, so then I ran describe and got this output
Normal Scheduled 16m default-scheduler Successfully assigned default/stock-api-7f5c45776b-gc67c to k3d-test-server-0
Normal Pulled 15m kubelet Successfully pulled image "mpriv32/stock-api:latest" in 16.509616605s
Normal Pulled 15m kubelet Successfully pulled image "mpriv32/stock-api:latest" in 696.527075ms
Normal Pulled 15m kubelet Successfully pulled image "mpriv32/stock-api:latest" in 734.334806ms
Normal Pulled 15m kubelet Successfully pulled image "mpriv32/stock-api:latest" in 823.429206ms
Normal Started 15m (x4 over 15m) kubelet Started container stock-api
Normal Pulling 14m (x5 over 16m) kubelet Pulling image "mpriv32/stock-api:latest"
Normal Pulled 14m kubelet Successfully pulled image "mpriv32/stock-api:latest" in 698.883126ms
Normal Created 14m (x5 over 15m) kubelet Created container stock-api
Warning BackOff 62s (x67 over 15m) kubelet Back-off restarting failed container
It constantly keeps "backoff restarting"
When I run `describe po` I get this
Name: stock-api-7f5c45776b-gc67c
Namespace: default
Priority: 0
Node: k3d-test-server-0/172.18.0.2
Start Time: Mon, 23 May 2022 06:44:42 0000
Labels: app=stock-api
pod-template-hash=7f5c45776b
Annotations: <none>
Status: Running
IP: 10.42.0.9
IPs:
IP: 10.42.0.9
Controlled By: ReplicaSet/stock-api-7f5c45776b
Containers:
stock-api:
Container ID: containerd://846d4c5c282274453c4b2ad8b834f20d2c673468ca18386d7404b07915b81a9c
Image: mpriv32/stock-api:latest
Image ID: docker.io/mpriv32/stock-api@sha256:98189cdf972ed61af79505b58aba2a0166fd012f5be4e0f012b2dffa0ea3dd5f
Port: <none>
Host Port: <none>
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Completed
Exit Code: 0
Started: Mon, 23 May 2022 08:23:16 0000
Finished: Mon, 23 May 2022 08:23:17 0000
Ready: False
Restart Count: 24
Environment Variables from:
api-credentials Secret Optional: false
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-czkv9 (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
kube-api-access-czkv9:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning BackOff 3m8s (x457 over 102m) kubelet Back-off restarting failed container
Dockerfile for my application
FROM python:3.8
COPY app.py .
RUN pip install requests python-dotenv
ARG API_KEY
ARG EMAIL_USER
ARG EMAIL_PASSWORD
ENV API_KEY $API_KEY
ENV EMAIL_USER $EMAIL_USER
ENV EMAIL_PASSWORD $EMAIL_PASSWORD
COPY database.py .
RUN pip install boto3
ARG ACCESS_KEY
ARG SECRET_KEY
ENV ACCESS_KEY $ACCESS_KEY
ENV SECRET_KEY $SECRET_KEY
CMD ["python", "database.py"]
EXPOSE 80
app.py
from datetime import datetime
import smtplib
import os
import requests
today_date = {datetime.today().strftime("%Y-%m-%d")}
url = (
'https://api.polygon.io/v1/open-close/{stock}/2022-05-10?adjusted=true&apiKey={API_key}'
)
if os.path.isfile('.env'):
from dotenv import load_dotenv
load_dotenv()
def __send_email(stock_data: str) -> None:
gmail_user = os.getenv('EMAIL_USER')
gmail_password = os.getenv('EMAIL_PASSWORD')
mail_from = gmail_user
mail_to = gmail_user
mail_subject = f'Your stock update for {datetime.today().strftime("%m/%d/%Y")}'
mail_message = f'Subject: {mail_subject}\n\n{stock_data}'
server = smtplib.SMTP('smtp.gmail.com', 587)
server.starttls()
server.login(gmail_user, gmail_password)
server.sendmail(mail_from, mail_to, mail_message)
server.close()
class api_data:
response = requests.get(url.format(
stock=os.getenv('stock'),
API_key=os.getenv('API_key')
))
data = response.json()
ticker = data['symbol']
daily_high = data['high']
daily_low = data['low']
daily_close = data['close']
def __email_body():
stock_data = (
f"""For the stock: {api_data.ticker}
\nToday's high was {api_data.daily_high}
\nToday's low was $ {api_data.daily_low}
\n{api_data.ticker} closed at $ {api_data.daily_close}"""
)
__send_email(stock_data)
__email_body()
database.py
import boto3
import app
import os
#Pulling stock name and daily price from API
stock_name = str(app.api_data.ticker)
stock_price = str(app.api_data.daily_close)
if os.path.isfile('.env'):
from dotenv import load_dotenv
load_dotenv()
ACCESS_KEY = os.getenv('ACCESS_KEY')
SECRET_KEY = os.getenv('SECRET_KEY')
#DynamoDB table name
table_name = "stock_tracker"
dynamodb_client = boto3.client('dynamodb', region_name='us-west-2', aws_access_key_id=ACCESS_KEY, aws_secret_access_key=SECRET_KEY)
stock_info ={
'Company': {'S': stock_name}
,'DailyPrice': {'S': stock_price}
}
print(stock_info)
if __name__ == "__main__":
dynamodb_client.put_item(TableName = table_name, Item = stock_info )
CodePudding user response：
Both your python files are executing batch actions.
The app.py file sends an email, the database.py stores something in a database.
That is not how Pods are supposed to work. Pods are meant to be persistent, or in other words to run continuously. Examples of workloads that are meant to run inside a pod are webservers or DBMSes.
For the two applications you showed us, you might want to use a Job instead (a Job runs your container once and than exits) or a CronJob maybe if you need your scripts to run multiple times on a schedule.
CodePudding user response：
first of all the below line doesnt appears to be a log entry
{'Company': {'S': 'AAPL'}, 'DailyPrice': {'S': '166.02'}}
you need to have log option, timestamp and other standard values in the log entry. It appears to be a code issue. plug-in standard logging lib like log4j and add log level like debug/warn/error etc to output proper log events. This would help you troubleshoot the issue.