About
Starter Tutorial
Uploaded 1 mo ago
by RobinHorton
- Object Notebook
Download
Starter Tutorial¶
This tutorial demonstrates basic steps for conducting an AI-powered survey using EDSL.
Prerequisites¶
Before running the code below, ensure that you have already completed technical setup:
Download the EDSL package. See Installation instructions.
Create a Coop account and activate Remote Inference or store your own API Keys for language models that you plan to use locally.
If you encounter any issues or have questions, please email us at info@expectedparrot.com or post a question at our Discord channel.
Conducting an AI-powered survey¶
In the steps below we show how to create and run a simple question in EDSL. Then we show how to design a more complex survey with AI agents and different language models.
Run a simple question¶
In this first example we create a simple multiple choice question, administer it to a language model and inspect the response:
In [1]:
# Import a question type
from edsl.questions import QuestionMultipleChoice
# Construct a question in the question type template
q = QuestionMultipleChoice(
question_name = "example_question",
question_text = "How do you feel today?",
question_options = ["Bad", "OK", "Good"]
)
# Run it with the default language model
results = q.run()
# Inspect the results
results.select("example_question").print()
| answer.example_question |
|---|
| Good |
Note: The default language model is currently GPT 4; you will need an API key for OpenAI to use this model and run this example locally. See instructions on storing your API Keys. Alternatively, you can activate Remote Inference at your Coop account to run the example on the Expected Parrot server.
Conduct a survey with agents and models¶
In this next example we create a survey of multiple questions, design personas for AI agents to answer the questions and select the language models that we want to generate the responses. We also show how to parameterize questions with context or data to administer different versions of the questions efficiently. This can be useful for comparing responses for different data, agents and models, and performing data labeling tasks.
We also show how to filter, sort, select and print components of the dataset of results.
To see examples of all EDSL question types, run:
In [2]:
from edsl import Question
Question.available()
Out[2]:
['budget', 'checkbox', 'extract', 'free_text', 'functional', 'likert_five', 'linear_scale', 'list', 'multiple_choice', 'numerical', 'rank', 'top_k', 'yes_no']
Newly released language models are automatically added to EDSL when they become available. To see a current list of available models, run:
In [3]:
from edsl import Model
Model.available()
Out[3]:
[['01-ai/Yi-34B-Chat', 'deep_infra', 0], ['Austism/chronos-hermes-13b-v2', 'deep_infra', 1], ['Gryphe/MythoMax-L2-13b', 'deep_infra', 2], ['Gryphe/MythoMax-L2-13b-turbo', 'deep_infra', 3], ['HuggingFaceH4/zephyr-orpo-141b-A35b-v0.1', 'deep_infra', 4], ['Phind/Phind-CodeLlama-34B-v2', 'deep_infra', 5], ['Qwen/Qwen2-72B-Instruct', 'deep_infra', 6], ['Qwen/Qwen2-7B-Instruct', 'deep_infra', 7], ['Sao10K/L3-70B-Euryale-v2.1', 'deep_infra', 8], ['bigcode/starcoder2-15b', 'deep_infra', 9], ['bigcode/starcoder2-15b-instruct-v0.1', 'deep_infra', 10], ['claude-3-5-sonnet-20240620', 'anthropic', 11], ['claude-3-haiku-20240307', 'anthropic', 12], ['claude-3-opus-20240229', 'anthropic', 13], ['claude-3-sonnet-20240229', 'anthropic', 14], ['codellama/CodeLlama-34b-Instruct-hf', 'deep_infra', 15], ['codellama/CodeLlama-70b-Instruct-hf', 'deep_infra', 16], ['cognitivecomputations/dolphin-2.6-mixtral-8x7b', 'deep_infra', 17], ['cognitivecomputations/dolphin-2.9.1-llama-3-70b', 'deep_infra', 18], ['databricks/dbrx-instruct', 'deep_infra', 19], ['deepinfra/airoboros-70b', 'deep_infra', 20], ['gemini-pro', 'google', 21], ['google/codegemma-7b-it', 'deep_infra', 22], ['google/gemma-1.1-7b-it', 'deep_infra', 23], ['google/gemma-2-27b-it', 'deep_infra', 24], ['google/gemma-2-9b-it', 'deep_infra', 25], ['gpt-3.5-turbo', 'openai', 26], ['gpt-3.5-turbo-0125', 'openai', 27], ['gpt-3.5-turbo-0301', 'openai', 28], ['gpt-3.5-turbo-0613', 'openai', 29], ['gpt-3.5-turbo-1106', 'openai', 30], ['gpt-3.5-turbo-16k', 'openai', 31], ['gpt-3.5-turbo-16k-0613', 'openai', 32], ['gpt-3.5-turbo-instruct', 'openai', 33], ['gpt-3.5-turbo-instruct-0914', 'openai', 34], ['gpt-4', 'openai', 35], ['gpt-4-0125-preview', 'openai', 36], ['gpt-4-0613', 'openai', 37], ['gpt-4-1106-preview', 'openai', 38], ['gpt-4-1106-vision-preview', 'openai', 39], ['gpt-4-turbo', 'openai', 40], ['gpt-4-turbo-2024-04-09', 'openai', 41], ['gpt-4-turbo-preview', 'openai', 42], ['gpt-4-vision-preview', 'openai', 43], ['gpt-4o', 'openai', 44], ['gpt-4o-2024-05-13', 'openai', 45], ['gpt-4o-mini', 'openai', 46], ['gpt-4o-mini-2024-07-18', 'openai', 47], ['lizpreciatior/lzlv_70b_fp16_hf', 'deep_infra', 48], ['llava-hf/llava-1.5-7b-hf', 'deep_infra', 49], ['meta-llama/Llama-2-13b-chat-hf', 'deep_infra', 50], ['meta-llama/Llama-2-70b-chat-hf', 'deep_infra', 51], ['meta-llama/Llama-2-7b-chat-hf', 'deep_infra', 52], ['meta-llama/Meta-Llama-3-70B-Instruct', 'deep_infra', 53], ['meta-llama/Meta-Llama-3-8B-Instruct', 'deep_infra', 54], ['meta-llama/Meta-Llama-3.1-405B-Instruct', 'deep_infra', 55], ['meta-llama/Meta-Llama-3.1-70B-Instruct', 'deep_infra', 56], ['meta-llama/Meta-Llama-3.1-8B-Instruct', 'deep_infra', 57], ['microsoft/Phi-3-medium-4k-instruct', 'deep_infra', 58], ['microsoft/WizardLM-2-7B', 'deep_infra', 59], ['microsoft/WizardLM-2-8x22B', 'deep_infra', 60], ['mistralai/Mistral-7B-Instruct-v0.1', 'deep_infra', 61], ['mistralai/Mistral-7B-Instruct-v0.2', 'deep_infra', 62], ['mistralai/Mistral-7B-Instruct-v0.3', 'deep_infra', 63], ['mistralai/Mixtral-8x22B-Instruct-v0.1', 'deep_infra', 64], ['mistralai/Mixtral-8x22B-v0.1', 'deep_infra', 65], ['mistralai/Mixtral-8x7B-Instruct-v0.1', 'deep_infra', 66], ['nvidia/Nemotron-4-340B-Instruct', 'deep_infra', 67], ['openchat/openchat-3.6-8b', 'deep_infra', 68], ['openchat/openchat_3.5', 'deep_infra', 69]]
In [4]:
# Import question types and survey components
from edsl import (
QuestionLinearScale, QuestionFreeText, Survey,
ScenarioList, Scenario,
AgentList, Agent,
ModelList, Model
)
# Construct questions
q1 = QuestionLinearScale(
question_name = "enjoy",
question_text = "On a scale from 1 to 5, how much do you enjoy {{ activity }}?",
question_options = [1,2,3,4,5],
option_labels = {1:"Not at all", 5:"Very much"}
)
q2 = QuestionFreeText(
question_name = "recent",
question_text = "Describe the most recent time you were {{ activity }}."
)
# Combine questions in a survey
survey = Survey(questions = [q1, q2])
# Add data to questions using scenarios
activities = ["exercising", "reading", "cooking"]
scenarios = ScenarioList(
Scenario({"activity": a}) for a in activities
)
# Create personas for AI agents to answer the questions
personas = ["athlete", "student", "chef"]
agents = AgentList(
Agent(traits = {"persona": p}) for p in personas
)
# Select language models to generate responses
models = ModelList(
Model(m) for m in ["gpt-4o", "claude-3-5-sonnet-20240620"]
)
# Run the survey with the scenarios, agents and models
results = survey.by(scenarios).by(agents).by(models).run()
# Filter, sort, select and print components of the results to inspect
(results
.filter("activity == 'reading' and persona == 'chef'")
.sort_by("model")
.select("model", "activity", "persona", "answer.*")
.print(format="rich",
pretty_labels = ({"model.model":"Model",
"scenario.activity":"Activity",
"agent.persona":"Agent persona",
"answer.enjoy":"Enjoy",
"answer.recent":"Recent"})
)
)
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Model ┃ Activity ┃ Agent persona ┃ Enjoy ┃ Recent ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ claude-3-5-sonnet-20240620 │ reading │ chef │ 4 │ As a chef, I recently found myself engrossed in │ │ │ │ │ │ a new cookbook featuring innovative │ │ │ │ │ │ Mediterranean cuisine. I was curled up in my │ │ │ │ │ │ favorite armchair, poring over vibrant photos │ │ │ │ │ │ of colorful dishes and reading about unique │ │ │ │ │ │ flavor combinations. The book's pages were │ │ │ │ │ │ filled with enticing recipes and culinary │ │ │ │ │ │ techniques that sparked my creativity. I spent │ │ │ │ │ │ hours taking mental notes, imagining how I │ │ │ │ │ │ could incorporate these ideas into my own │ │ │ │ │ │ cooking. Reading cookbooks is not just a │ │ │ │ │ │ pastime for me; it's an essential part of my │ │ │ │ │ │ professional development and a source of │ │ │ │ │ │ endless inspiration in the kitchen. │ ├────────────────────────────┼──────────┼───────────────┼───────┼─────────────────────────────────────────────────┤ │ gpt-4o │ reading │ chef │ 4 │ The most recent time I was reading, I was │ │ │ │ │ │ diving into a new cookbook that focuses on │ │ │ │ │ │ Mediterranean cuisine. I was particularly │ │ │ │ │ │ fascinated by the section on traditional Greek │ │ │ │ │ │ dishes. The vibrant descriptions of fresh │ │ │ │ │ │ ingredients like olives, feta cheese, and herbs │ │ │ │ │ │ transported me straight to the sun-drenched │ │ │ │ │ │ coasts of Greece. I couldn't wait to try out │ │ │ │ │ │ the recipe for Moussaka; the layers of │ │ │ │ │ │ eggplant, minced meat, and béchamel sauce │ │ │ │ │ │ sounded irresistible! │ └────────────────────────────┴──────────┴───────────────┴───────┴─────────────────────────────────────────────────┘
Exploring your results¶
EDSL comes with built-in methods for analyzing and visualizing your results. For example, you can access results as a Pandas dataframe:
In [5]:
# Convert the Results object to a pandas dataframe
(results
.sort_by("model", "activity", "persona")
.select("model", "activity", "persona", "recent", "enjoy")
.to_pandas(remove_prefix=True)
)
Out[5]:
| activity | enjoy | model | persona | recent | |
|---|---|---|---|---|---|
| 0 | cooking | 3 | claude-3-5-sonnet-20240620 | athlete | As an athlete, I don't spend much time cooking... |
| 1 | cooking | 5 | claude-3-5-sonnet-20240620 | chef | The most recent time I was cooking was just ye... |
| 2 | cooking | 3 | claude-3-5-sonnet-20240620 | student | The most recent time I was cooking was last ni... |
| 3 | exercising | 5 | claude-3-5-sonnet-20240620 | athlete | The most recent time I was exercising was this... |
| 4 | exercising | 4 | claude-3-5-sonnet-20240620 | chef | As a chef, I don't typically have a lot of tim... |
| 5 | exercising | 3 | claude-3-5-sonnet-20240620 | student | The most recent time I exercised was yesterday... |
| 6 | reading | 3 | claude-3-5-sonnet-20240620 | athlete | As an athlete, I don't spend a lot of time rea... |
| 7 | reading | 4 | claude-3-5-sonnet-20240620 | chef | As a chef, I recently found myself engrossed i... |
| 8 | reading | 4 | claude-3-5-sonnet-20240620 | student | The most recent time I was reading was last ni... |
| 9 | cooking | 4 | gpt-4o | athlete | The most recent time I was cooking, I decided ... |
| 10 | cooking | 5 | gpt-4o | chef | The most recent time I was cooking, I was prep... |
| 11 | cooking | 4 | gpt-4o | student | The most recent time I was cooking, I made a s... |
| 12 | exercising | 5 | gpt-4o | athlete | The most recent time I was exercising was yest... |
| 13 | exercising | 4 | gpt-4o | chef | The most recent time I was exercising was just... |
| 14 | exercising | 4 | gpt-4o | student | The most recent time I was exercising was yest... |
| 15 | reading | 3 | gpt-4o | athlete | The most recent time I was reading, I was goin... |
| 16 | reading | 4 | gpt-4o | chef | The most recent time I was reading, I was divi... |
| 17 | reading | 5 | gpt-4o | student | The most recent time I was reading was last ni... |
The columns method will display a list of all the components of your results, which you can then select and print to show them:
In [6]:
results.columns
Out[6]:
['agent.agent_instruction', 'agent.agent_name', 'agent.persona', 'answer.enjoy', 'answer.recent', 'comment.enjoy_comment', 'iteration.iteration', 'model.frequency_penalty', 'model.logprobs', 'model.max_tokens', 'model.model', 'model.presence_penalty', 'model.temperature', 'model.top_logprobs', 'model.top_p', 'prompt.enjoy_system_prompt', 'prompt.enjoy_user_prompt', 'prompt.recent_system_prompt', 'prompt.recent_user_prompt', 'question_options.enjoy_question_options', 'question_options.recent_question_options', 'question_text.enjoy_question_text', 'question_text.recent_question_text', 'question_type.enjoy_question_type', 'question_type.recent_question_type', 'raw_model_response.enjoy_raw_model_response', 'raw_model_response.recent_raw_model_response', 'scenario.activity']
The Results object also supports SQL-like queries:
In [7]:
# Execute an SQL-like query on the results
results.sql("""
select model, activity, persona, recent, enjoy
from self
order by 1,2,3
""", shape="wide")
Out[7]:
| model | activity | persona | recent | enjoy | |
|---|---|---|---|---|---|
| 0 | claude-3-5-sonnet-20240620 | cooking | athlete | As an athlete, I don't spend much time cooking... | 3 |
| 1 | claude-3-5-sonnet-20240620 | cooking | chef | The most recent time I was cooking was just ye... | 5 |
| 2 | claude-3-5-sonnet-20240620 | cooking | student | The most recent time I was cooking was last ni... | 3 |
| 3 | claude-3-5-sonnet-20240620 | exercising | athlete | The most recent time I was exercising was this... | 5 |
| 4 | claude-3-5-sonnet-20240620 | exercising | chef | As a chef, I don't typically have a lot of tim... | 4 |
| 5 | claude-3-5-sonnet-20240620 | exercising | student | The most recent time I exercised was yesterday... | 3 |
| 6 | claude-3-5-sonnet-20240620 | reading | athlete | As an athlete, I don't spend a lot of time rea... | 3 |
| 7 | claude-3-5-sonnet-20240620 | reading | chef | As a chef, I recently found myself engrossed i... | 4 |
| 8 | claude-3-5-sonnet-20240620 | reading | student | The most recent time I was reading was last ni... | 4 |
| 9 | gpt-4o | cooking | athlete | The most recent time I was cooking, I decided ... | 4 |
| 10 | gpt-4o | cooking | chef | The most recent time I was cooking, I was prep... | 5 |
| 11 | gpt-4o | cooking | student | The most recent time I was cooking, I made a s... | 4 |
| 12 | gpt-4o | exercising | athlete | The most recent time I was exercising was yest... | 5 |
| 13 | gpt-4o | exercising | chef | The most recent time I was exercising was just... | 4 |
| 14 | gpt-4o | exercising | student | The most recent time I was exercising was yest... | 4 |
| 15 | gpt-4o | reading | athlete | The most recent time I was reading, I was goin... | 3 |
| 16 | gpt-4o | reading | chef | The most recent time I was reading, I was divi... | 4 |
| 17 | gpt-4o | reading | student | The most recent time I was reading was last ni... | 5 |
In [8]:
results.push(visibility="public")
Out[8]:
{'description': None,
'object_type': 'results',
'url': 'https://www.expectedparrot.com/content/d36ba048-e401-422e-9394-6101fe413b8d',
'uuid': 'd36ba048-e401-422e-9394-6101fe413b8d',
'version': '0.1.30.dev3',
'visibility': 'public'}
Posting this notebook:
In [9]:
from edsl import Coop, Notebook
coop = Coop()
notebook = Notebook(path="starter_tutorial.ipynb")
coop.create(notebook, description="Starter Tutorial", visibility="public")
Out[9]:
{'description': None,
'object_type': 'notebook',
'url': 'https://www.expectedparrot.com/content/e4f911be-030e-44bc-a2a8-8a65c4b7b061',
'uuid': 'e4f911be-030e-44bc-a2a8-8a65c4b7b061',
'version': '0.1.30.dev3',
'visibility': 'public'}
from edsl import Notebook
notebook = Notebook.pull("6e6508ca-353e-41f7-866e-7a3c6232e0fd")
# this object is too large to display here