---
title: Generative AI Study Group_15thSesssion_20240116
tags:  #ai #generative ai #machine learning #deep learning #artificial intelligence  
author: [Kunihiro Sugiyama](https://convert.docswell.com/user/KunihiroSugiyama)
site: [Docswell](https://www.docswell.com/)
thumbnail: https://bcdn.docswell.com/page/8EDD6W46EG.jpg?width=480
description: 2024年生成系AIの展望と考察  こんにちわ! GASG 主催の杉山です。 https://www.ai-tech-c.jp/generative-ai-study-group-gasg/ 第15回となります。  さて、今回2024年のGen-AIを展望する新春企画です。 2023年大きく発展飛躍したGen-AIが今年どのように変化していくか。 様々な角度から今年一年のトレンドを読み解いていきたいと思います。 ぜひお楽しみに!!
published: January 24, 24
canonical: https://convert.docswell.com/s/KunihiroSugiyama/KENDL7-2024-01-24-192336
---
# Page. 1

![Page Image](https://bcdn.docswell.com/page/8EDD6W46EG.jpg)

Generative
AI Study
Group


# Page. 2

![Page Image](https://bcdn.docswell.com/page/V7PXY9VZE8.jpg)

Agenda


# Page. 3

![Page Image](https://bcdn.docswell.com/page/2JVLGMZM7Q.jpg)

Agenda


# Page. 4

![Page Image](https://bcdn.docswell.com/page/5EGX8G4XJL.jpg)

Introduction
• 今日のパネラー紹介
• 勉強会貢献メンバー


# Page. 5

![Page Image](https://bcdn.docswell.com/page/4JQ89XG57P.jpg)

Introduction
How make AI do it
is all you need!!


# Page. 6

![Page Image](https://bcdn.docswell.com/page/K742X65VJ1.jpg)

Introduction
Yes, AI can!!


# Page. 7

![Page Image](https://bcdn.docswell.com/page/LJ1MNVW4EG.jpg)

Introduction
• 公式X
• https://twitter.com/kunihirosks


# Page. 8

![Page Image](https://bcdn.docswell.com/page/GJW34L6ZJ2.jpg)

Introduction
• 前回の振り返り
• Web更新
• https://www.ai-tech-c.jp/generative-ai-study-group-gasg/
• アンケート結果
• 今回のアンケート


# Page. 9

![Page Image](https://bcdn.docswell.com/page/4EZV2MYLJ3.jpg)

Introduction
•進捗
• テーマとゲスト急募!!
• 新企画
• セキュリティ
• アルファブロガー hoge様


# Page. 10

![Page Image](https://bcdn.docswell.com/page/Y765VNDMEV.jpg)

Introduction
• 参加者名簿
• 個別にご連絡をさせていただくため活用させていただく場合があ
りますので、是非。
• Form
• https://forms.gle/eRmeGtfkQMkDrs7K9


# Page. 11

![Page Image](https://bcdn.docswell.com/page/G75YD53Q74.jpg)

Introduction
•AITeCに是非ご入会をご検討ください
•驚きの安さ!!
•なんと年間10万円ポッキリ!!
•安い! 楽しい! 凄い! AITeC!!


# Page. 12

![Page Image](https://bcdn.docswell.com/page/9J2G8NZWJR.jpg)

Introduction
• AITeC会員限定
•
Mattermost
•
•
情報 / 企画 / ビジョン
•
厳選とれたて新鮮Info from bookmarks
•
僕が一番ChatGPTをうまくつかえるんだぁー!!大会
•
ChatGPT相談室
•
疑似人格
•
AIを使った社会実験
•
Generative Agents
•
新AIパラダイム時代の生き方論
•
Deparure of Computable Fundamentalism
ABCI利用
•
スーパーエンジニアが親切にサポート (予定


# Page. 13

![Page Image](https://bcdn.docswell.com/page/DEYV6PR9EM.jpg)

Introduction
• AITeCオープンイベント
o【ABCIでLLMを動かそう！セミナー 第2回】 機械学
習・ディープラーニングとは - 人工知能技術コンソー
シアム | Doorkeeper
https://aitconsortium.doorkeeper.jp/events/168119


# Page. 14

![Page Image](https://bcdn.docswell.com/page/VJNMZ5DD78.jpg)

Agenda


# Page. 15

![Page Image](https://bcdn.docswell.com/page/YE9NZYG8E3.jpg)

Theme
• Title
• NISTEPによる生成AIとDXのすがた
• Presenter
• 盛岡広域振興局 佐藤 清忠氏
• Contents
• https://drive.google.com/file/d/1kU6PVDymrcyABr59ag90Nb4VpywZ0WB/view?usp=sharing


# Page. 16

![Page Image](https://bcdn.docswell.com/page/GE8K16VZ7D.jpg)

Theme
• Title
• 2024年生成系AIの展望と考察
▪ Mainly NLP topics
• Presenter
• 杉山邦洋 @Generative Ai Study Group .host
• Contents
• Tech trend, Use case, Issue


# Page. 17

![Page Image](https://bcdn.docswell.com/page/LELZQ951JR.jpg)

Theme
•「マルチモーダルAI」「小規模言語モデ
ル」2024年の生成AI重要トレンド（Forbes
JAPAN） - Yahoo!ニュース
https://news.yahoo.co.jp/articles/633019417533
9a101c2f6e8ad36b0c4109943939?page=1


# Page. 18

![Page Image](https://bcdn.docswell.com/page/4JMV1Z35JW.jpg)

Tech trend


# Page. 19

![Page Image](https://bcdn.docswell.com/page/PJR2L6QZE9.jpg)

Tech trend
•Small model
•Beyond Transformer
•Related tech


# Page. 20

![Page Image](https://bcdn.docswell.com/page/PEXDWM517X.jpg)

Tech trend
• Small model
• Open LLM Leaderboard - a Hugging Face Space by HuggingFaceH4
https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard
• Points
▪ コンピューティング量
▪ 実行メモリ量
▪ 推論速度
▪ 特化型カスタマイゼーション


# Page. 21

![Page Image](https://bcdn.docswell.com/page/3EKXVG6M7D.jpg)

Tech trend
• Small model
• Pickup
▪ TinyLlama
• jzhang38/TinyLlama:The TinyLlama project is an open endeavor to pretrain a 1.1B Llama
model on 3 trillion tokens. https://github.com/jzhang38/TinyLlama
• 【軽量かつ高速なLLM】TinyLlamaについてまとめてみた #LLM - Qiita
https://qiita.com/sergicalsix/items/7cd7665ab90b9f3b343c
• LLAMAと完全互換のアーキテクチャおよびトークンナイザー
• 1.1Bパラメータモデルは4bit量子化でおよそ550MB RAM上で動作


# Page. 22

![Page Image](https://bcdn.docswell.com/page/L735MN62E5.jpg)

Tech trend
• Small model
• Pickup
▪ Phi-2
• Phi-2: The surprising power of small language models - Microsoft Research
https://www.microsoft.com/en-us/research/blog/phi-2-the-surprising-powerof-small-language-models/
• [2306.11644] Textbooks Are All You Need https://arxiv.org/abs/2306.11644
• 2.7Bパラメータモデル
• 高品質な学習データセットでモデル品質を確保


# Page. 23

![Page Image](https://bcdn.docswell.com/page/87DD6WV67G.jpg)

Tech trend
• Small model
• Pickup
▪ Orca 2
• Orca - Microsoft Research https://www.microsoft.com/en-us/research/project/orca/
• Microsoft&#039;s Orca 2 LLM Outperforms Models That Are 10x Larger
https://www.infoq.com/news/2023/12/microsoft-orca-2-llm/
• 7B, 13Bパラメータモデル
• LLAMA-2のFinetunedモデル
• Reasoningが含まれる合成データセットで訓練


# Page. 24

![Page Image](https://bcdn.docswell.com/page/VJPXY9RZ78.jpg)

Tech trend
• Small model
• Pickup
▪ DeciLM
• Deci/DeciLM-7B · Hugging Face https://huggingface.co/Deci/DeciLM-7B
• [2305.13245] GQA:Training Generalized Multi-Query Transformer Models from Multi-Head
Checkpoints https://arxiv.org/abs/2305.13245
• What is Grouped Query Attention (GQA)? — Klu https://klu.ai/glossary/grouped-queryattention
• GQAの仕組みを採用し軽量で高性能
• GQAは例えばLLaMA2 70Bでも使われている


# Page. 25

![Page Image](https://bcdn.docswell.com/page/2EVLGMRMJQ.jpg)

Tech trend
二次的複雑性 (Quadratic complexity O(n^2)) http://tinyurl.com/yqs8pwec
• Beyond Transformer
• Points
▪ Transformerの課題解決の試み
• 推論速度
• 実行メモリ量
• シーケンス長
• コンピューティング量


# Page. 26

![Page Image](https://bcdn.docswell.com/page/57GX8GMXEL.jpg)

Tech trend
• Beyond Transformer
• Pickup
▪ MoE (Mixture of Experts)
Reference: http://tinyurl.com/ylxsvomj


# Page. 27

![Page Image](https://bcdn.docswell.com/page/4EQ89XR5JP.jpg)

Tech trend
• Beyond Transformer
• Pickup
▪ MoE (Mixture of Experts)
• 論文解説 Outrageously Large Neural Networks: The Sparsely-Gated
Mixture-of-Experts Layer (MoE) - ディープラーニングブログ
https://deeplearning.hatenablog.com/entry/moe
• Mixture of expertsの簡単な実装をしてみる
https://zenn.dev/if001/articles/40917524959913
• MixtralSparseMoeBlockを読む https://zenn.dev/if001/articles/fcea9fe9f1bdb1


# Page. 28

![Page Image](https://bcdn.docswell.com/page/KJ42X68VE1.jpg)

Tech trend
• Beyond Transformer
• Pickup
▪ RWKV (Reinventing RNNs for the Transformer Era)
• RWKVについて解説 | AGIRobots Blog
https://developers.agirobots.com/jp/rwkv/
• RWKVを論文と実装から読み解く https://zenn.dev/jow/articles/f66d6403b9a509
• RNNでTransformer並みの性能を実現するRWKVがやばい
https://zenn.dev/hikettei/articles/5d6c1318998411
• これは衝撃!1.5Bで超高性能LLM!RWKV-5-World-v2｜shi3z
https://note.com/shi3zblog/n/nfc8dd1abf494


# Page. 29

![Page Image](https://bcdn.docswell.com/page/LE1MNV247G.jpg)

Tech trend
• Beyond Transformer
• Pickup
▪ Mamba
• state-spaces/mamba https://github.com/state-spaces/mamba
• [2312.00752] Mamba: Linear-Time Sequence Modeling with Selective State Spaces https://arxiv.org/abs/2312.00752
• Mamba: Redefining Sequence Modeling and Outforming Transformers Architecture - Unite.AI https://www.unite.ai/mambaredefining-sequence-modeling-and-outforming-transformers-architecture/
• Mamba: Linear-Time Sequence Modeling with Selective State Spaces — Arxiv Dives | by Oxen | Dec, 2023 | Medium
https://medium.com/@oxenai/mamba-linear-time-sequence-modeling-with-selective-state-spaces-arxiv-dives-cf96518d7ec4
▪ StripedHyena
• Architectures for longer sequences and efficient inference: StripedHyena | hessian.AI https://hessian.ai/architectures-for-longersequences-and-efficient-inference-stripedhyena/
• [2302.10866] Hyena Hierarchy:Towards Larger Convolutional Language Models https://arxiv.org/abs/2302.10866


# Page. 30

![Page Image](https://bcdn.docswell.com/page/GEW34LRZE2.jpg)

Tech trend
• Beyond Transformer
• Pickup
▪ MoE-Mamba
• [2401.04081] MoE-Mamba: Efficient Selective State Space Models
with Mixture of Experts https://arxiv.org/abs/2401.04081
▪ Monarch Mixer
• [2310.12109] Monarch Mixer: A Simple Sub-Quadratic GEMMBased Architecture https://arxiv.org/abs/2310.12109


# Page. 31

![Page Image](https://bcdn.docswell.com/page/47ZV2MRLE3.jpg)

Tech trend
•Related tech
•RAG (Retrieval-Augmented Generation)
•Agent
•Synthetic data (合成データ)
•Distributed


# Page. 32

![Page Image](https://bcdn.docswell.com/page/YJ65VNRM7V.jpg)

Tech trend
• Related tech
• RAG (Retrieval-Augmented Generation)
▪第6回で取り上げました
▪進化するRAGアーキテクチャ
▪[2312.10997] Retrieval-Augmented Generation for Large
Language Models:A Survey https://arxiv.org/abs/2312.10997


# Page. 33

![Page Image](https://bcdn.docswell.com/page/GJ5YD58QJ4.jpg)

Reference: https://arxiv.org/pdf/2312.10997.pdf
Figure 6: RAG compared with other model optimization methods
Theme
• Tech Trend
• Related tech
▪ RAG (Retrieval-Augmented Generation)


# Page. 34

![Page Image](https://bcdn.docswell.com/page/LE35MN8ZJ5.jpg)

Reference: https://arxiv.org/pdf/2312.10997.pdf
Figure 2: A representative instance of the RAG process applied to question answering
Theme
• Tech Trend
• Related tech
▪ RAG (Retrieval-Augmented Generation)


# Page. 35

![Page Image](https://bcdn.docswell.com/page/8EDD6WM4EG.jpg)

Reference: https://arxiv.org/pdf/2312.10997.pdf
Figure 3: Comparison between the three paradigms of RAG
Theme
• Tech Trend
• Related tech
▪ RAG (Retrieval-Augmented Generation)


# Page. 36

![Page Image](https://bcdn.docswell.com/page/V7PXY9RVE8.jpg)

Tech trend
• Related tech
• RAG (Retrieval-Augmented Generation)
▪Advanced
• Optimizing data indexing
• Pre retrieval process
• Post retrieval process


# Page. 37

![Page Image](https://bcdn.docswell.com/page/2JVLGMRR7Q.jpg)

Tech trend
• Related tech
• RAG (Retrieval-Augmented Generation)
▪ Modular
• 多様な機能モジュール
• ニーズに適したPipeline


# Page. 38

![Page Image](https://bcdn.docswell.com/page/5EGX8GM6JL.jpg)

Tech trend
• Related tech
• RAG (Retrieval-Augmented Generation)
▪ A Cheat Sheet and Some Recipes For Building Advanced RAG | by Andrei | Jan,
2024 | LlamaIndex Blog https://blog.llamaindex.ai/a-cheat-sheet-and-somerecipes-for-building-advanced-rag-803a9d94c41b


# Page. 39

![Page Image](https://bcdn.docswell.com/page/4JQ89XR27P.jpg)

Tech trend
• Related tech
• RAG (Retrieval-Augmented Generation)
▪ Scaling context window
▪ Robustness
• Hallucination
▪ Hybrid (RAG+FT)
▪ Expanding LLM role
▪ Scaling law
• Embedding model
▪ Production ready
• 精度, 再現性, セキュリティ(アクセスコントロール)
▪ Multi modal
• Image, Audio and video, Code


# Page. 40

![Page Image](https://bcdn.docswell.com/page/K742X68PJ1.jpg)

Tech trend
• Related tech
• Agent
▪ 第11回で取り上げました
Reference: https://medium.com/scisharp/understand-the-llm-agent-orchestration-043ebfaead1f


# Page. 41

![Page Image](https://bcdn.docswell.com/page/LJ1MNV2XEG.jpg)

Tech trend
• Related tech
• Agent
▪ 2024 AI Agent
▪ https://e2b.dev/blog/ai-agents-in-2024
▪ 評価フレームワーク
• THUDM/AgentBench:A Comprehensive Benchmark to Evaluate LLMs as Agents
https://github.com/THUDM/AgentBench
• AutoGPT/benchmark at master · Significant-Gravitas/AutoGPT
https://github.com/Significant-Gravitas/AutoGPT/tree/master/benchmark
• Benchmarking Agent Tool Use https://blog.langchain.dev/benchmarking-agent-tool-use


# Page. 42

![Page Image](https://bcdn.docswell.com/page/GJW34LRKJ2.jpg)

Tech trend
• Related tech
• 合成データ
▪ Synthetic data:Anthropic’s CAI, scaling, OpenAI’s Superalignment, tips, and open-source
examples https://www.interconnects.ai/p/llm-synthetic-data
▪ [2305.15041] Generating Faithful Synthetic Data with Large Language Models: A Case Study in
Computational Social Science https://arxiv.org/abs/2305.15041
▪ [2310.07849] Synthetic Data Generation with Large Language Models for Text Classification:
Potential and Limitations https://arxiv.org/abs/2310.07849
▪ [2401.00368] Improving Text Embeddings with Large Language Models
https://arxiv.org/abs/2401.00368
▪ [2312.17742] Learning Vision from Models Rivals Learning Vision from Data
https://arxiv.org/abs/2312.17742


# Page. 43

![Page Image](https://bcdn.docswell.com/page/4EZV2MRNJ3.jpg)

Tech trend
• Related tech
• Distributed
▪ Petals
• Petals – Run LLMs at home, BitTorrent-style https://petals.dev/
• bigscience-workshop/petals:
Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster
than offloading https://github.com/bigscience-workshop/petals
• [2209.01188] Petals: Collaborative Inference and Fine-tuning of Large Models https://arxiv.org/abs/2209.01188
• [2312.08361] Distributed Inference and Fine-tuning of Large Language Models Over The Internet
https://arxiv.org/abs/2312.08361
▪ AI Horde
• AI Horde https://stablehorde.net/
• Haidra-Org/AI-Horde: A crowdsourced distributed cluster for AI art and text generation
https://github.com/Haidra-Org/AI-Horde?tab=readme-ov-file


# Page. 44

![Page Image](https://bcdn.docswell.com/page/Y765VNR9EV.jpg)

Use case


# Page. 45

![Page Image](https://bcdn.docswell.com/page/G75YD58D74.jpg)

Use case
• 企業における生成AIの未来：ChatGPTを越えてその先へ |
ガートナー https://www.gartner.co.jp/ja/articles/beyond-chatgptthe-future-of-generative-ai-for-enterprises
• Top 100+ Generative AI Applications / Use Cases in 2024
https://research.aimultiple.com/generative-ai-applications/
• 2024 AI Predictions | NVIDIA Blog
https://blogs.nvidia.com/blog/2024-ai-predictions/


# Page. 46

![Page Image](https://bcdn.docswell.com/page/9J2G8NVMJR.jpg)

Use case
• Device
▪ Order Ai Pin Now https://hu.ma.ne/
▪ rabbit — home https://www.rabbit.tech/
▪ Avi Schiffmann’s Tab AI necklace raised $1.9 million to replace God
https://www.fastcompany.com/91007630/avi-schiffmanns-tab-ai-necklace-hasraised-1-9-million-to-replace-god
▪ OpenAI、「AI版iPhone」をアップル元デザイナーと開発か 孫正義に
出資打診 | Forbes JAPAN 公式サイト（フォーブス ジャパン）
https://forbesjapan.com/articles/detail/66348


# Page. 47

![Page Image](https://bcdn.docswell.com/page/DEYV6P3PEM.jpg)

Issue


# Page. 48

![Page Image](https://bcdn.docswell.com/page/VJNMZ5VM78.jpg)

Issue
•Security
•Data contamination
•Socialization


# Page. 49

![Page Image](https://bcdn.docswell.com/page/YE9NZY6WE3.jpg)

Issue
• Security
• Overview
▪ 8 Generative AI Security Risks That You Should Know https://www.globalsign.com/en/blog/8-generative-ai-security-risks
▪ Safety and security risks of generative artificial intelligence to 2025 (Annex B) - GOV.UK
https://www.gov.uk/government/publications/frontier-ai-capabilities-and-risks-discussion-paper/safety-and-security-risks-of-generativeartificial-intelligence-to-2025-annex-b
• Prompt hack
• GPTs のプロンプトリーキング対策｜ぬこぬこ https://note.com/schroneko/n/n6d6c2e645119
•
Prompt Hacking | Learn Prompting:Your Guide to Communicating with AI https://learnprompting.org/docs/category/-prompt-hacking
• Solution
▪ Introducing Purple Llama for Safe and Responsible AI Development | Meta https://about.fb.com/news/2023/12/purple-llama-saferesponsible-ai-development/
▪ New generative AI-powered SaaS security expert from AppOmni | VentureBeat https://venturebeat.com/security/new-generative-aipowered-saas-security-expert-from-appomni


# Page. 50

![Page Image](https://bcdn.docswell.com/page/GE8K16ZR7D.jpg)

Issue
• Data contamination
• Why data contamination is a big issue for LLMs - TechTalks
https://bdtechtalks.com/2023/07/17/llm-data-contamination/
• [2312.16337] Task Contamination: Language Models May Not Be Few-Shot Anymore
https://arxiv.org/abs/2312.16337
• Socialization
• Sotopia https://www.sotopia.world/
• [2310.11667] SOTOPIA: Interactive Evaluation for Social Intelligence in Language
Agents https://arxiv.org/abs/2310.11667


# Page. 51

![Page Image](https://bcdn.docswell.com/page/LELZQ9D2JR.jpg)

Agenda


# Page. 52

![Page Image](https://bcdn.docswell.com/page/4JMV1ZP9JW.jpg)

Study contents
• Coming Sooooooooooon!!


# Page. 53

![Page Image](https://bcdn.docswell.com/page/PJR2L619E9.jpg)

Agenda


# Page. 54

![Page Image](https://bcdn.docswell.com/page/PEXDWMY37X.jpg)

News
• みんなで生成系AIブックマーク
• 新企画です!! 皆様の気になった生成系AIの情報をその場で募集して
ディスカッションします。
• 募集はこちら https://forms.gle/kKVsb9jDwMTm2Rkk6 から事前投稿も受
け付けています。
• リスト
• https://docs.google.com/spreadsheets/d/1bjJfAhfEvNMSuxAszCiSo0qdE89s
p86ooAyBdRVuDO0/edit?usp=sharing


# Page. 55

![Page Image](https://bcdn.docswell.com/page/3EKXVG3N7D.jpg)

Agenda


# Page. 56

![Page Image](https://bcdn.docswell.com/page/L735MNGZE5.jpg)

Next
• アンケートの結果とコメント
• 次回
•
みんなで生成系AIブックマーク&quot; 継続
•
分析してみる？
•
Theme引き続き募集
•
Study contents
• セキュリティ
• Fine tuning
• ChatGPT Plugins
• オープンソース各種
• Langchain,LlamaIndex, 他
• Business向けコンテンツ
• New era personal computer


# Page. 57

![Page Image](https://bcdn.docswell.com/page/87DD6W147G.jpg)

EOF
https://www.linkedin.com/in/kunihiro-sugiyama-49b0372a/
https://www.ai-tech-c.jp/generative-ai-study-group-gasg/


