Prompt help – ChatGPT is just REPEATEDLY getting it wrong!

Optimizing Shift and Earnings Calculations Using AI: A Developer’s Experience with Prompt Engineering Challenges

In the quest to streamline payroll and shift tracking, many professionals turn to AI tools like ChatGPT for assistance. However, even advanced AI models can encounter significant hurdles, especially when it comes to interpreting visual data and adhering to complex instructions. I’d like to share my recent experience, highlighting the challenges and considerations for developers and users aiming to leverage AI effectively within WordPress or other platforms.

The Challenge: Accurate Data Extraction and Processing from Images

Over the past hour, I’ve been experimenting with ChatGPT to automate the calculation of my work shifts and earnings based on a photographic roster. Despite multiple attempts and refining my prompts, the AI persistently misreads essential details such as dates and days. Even after explicitly instructing it to interpret the roster layout as Monday through Sunday, it consistently errs—confusing the order, misassigning dates, or misunderstanding shift timings.

This issue underscores a common difficulty: AI language models are primarily designed for language understanding, not image recognition. While some features like GPT combined with image processing models exist, integrating them seamlessly within a WordPress environment remains complex.

Refining Prompt Design: A Recurrent Attempt

Here is an example of the prompt structure I’ve been using each time, intended to guide the AI accurately:

Shift and Pay Calculation Instructions

Please analyze the uploaded image of my weekly roster to compute my bi-weekly pay, adhering to these guidelines:

• Calendar Layout: Always interpret the roster as starting on Monday and ending on Sunday. Do not treat the week as Sunday to Saturday.
• Date Labels: Read and utilize the exact date labels as they appear on the image. Do not infer dates based on position or assumptions—extract the information directly from the image.
• Shift Colors:
• Orange: Morning shift (6 am – 2 pm)
• Blue: Afternoon shift (2 pm – 10 pm)
• Red: Night shift (10 pm – 6 am)
• Employment Details:
• Casual employee
• Base rate: $(XX.XX)/hour including casual loading
• Penalty Rates:
• 15% for afternoon shifts (12 pm – 6 pm)
• 15% for night shifts (10 pm – 6 am)
• 50% penalty for Saturday shifts
• 100% penalty for Sunday shifts