Forem: MongoDB Builders

🌐 开始使用: 什么是MongoDB操作数据层? (第2部分) 🌐

Danny Chan — Wed, 21 Aug 2024 14:21:40 +0000

操作数据层数据加载:
✅ 数据必须与源系统同步
✅ 适当的数据加载策略
✅ 生产系统: 数据变化的频率和数量
✅ 消费系统: 对数据时效性的明确要求

第1步: 批量提取和加载:
📁 初始批量加载
📁 将应用程序数据库数据复制到操作数据层
📁 一次性从源系统加载数据

第2步: 增量提取和加载:
🔄 紧跟初始批量加载之后立即开始
🔄 实时同步
🔄 从源系统增量更新到ODL
🔄 使用变更数据捕获(CDC)
🔄 捕获源系统的变更
🔍 匹配、合并、校准数据

数据流和成熟度模型:
🏗️ 起步简单
🌱 范围和战略重要性不断扩大
🏆 为业务带来越来越多的收益

第1阶段: 简单的ODL, 卸载读取:
💻 仅提供读取操作
💰 降低成本
🔒 高可用性: 在源系统宕机时接管
⚡ 提高性能
📊 处理长时间运行的分析查询
📈 应对高峰读取量

第2阶段: 功能丰富的ODL应用新场景:
🔍 建立单一的客户视图
💳 例如: 信用卡交易数据细化为各类消费
💰 分析每个消费类别(如旅行)的支出

第3阶段: 卸载读写:
📥 同时降低延迟,并行引入新系统

第4阶段: ODL优先:
✍️ 所有写入直接到操作数据层(ODL)

第5阶段: 系统的记录:
🗃️ 操作数据层成为系统的记录
🏳️ 源系统可以退役以节省成本
🏛️ 架构更加简单

MongoDB用于操作数据层:
🤗 便利性: MongoDB的文档模型易于管理数据
🧩 灵活性: 将多个源系统集成到单一ODL, 无需预先定义模式
⚡ 速度: 访问数据时性能更好
🎨 多功能性: 凭借文档模型的灵活性满足各种应用需求

示例:
📁 嵌套数组和子文档
📊 建模复杂关系和层次数据
🗄️ 无需重写整个文档即可操作深层嵌套数据
🗂️ 建模平面、类似表格的结构, 简单键值对, 文本
🌍 地理空间数据
🕸️ 图处理中使用的节点和边

处理管道:
🔍 查找和范围查询
📊 数据分析
🔨 转换
🔍 分面搜索
🌎 地理空间处理
🕵️ 图遍历

智能分布(操作数据层)ODL:

可用性:
💻 使用副本集保存多份数据副本
🔄 故障转移和恢复完全自动化

可扩展性:
🆙 挑战: 新的源系统, 数据量增加, 新的消费系统, 负载增大
🗄️ 大数据集
⚡ 高吞吐量需求
🧩 解决方案 - 分片:
🤖 MongoDB提供低成本的水平扩展
🔁 自动将数据分区并分布到多个物理实例

工作负载隔离:
🔍 操作数据层能够安全地服务于不同的工作负载
🔍 在不影响生产应用的情况下, 对最新数据进行分析查询

数据本地化:
🌎 精确控制数据实际存储位置
🗺️ 控制地理区域以满足延迟和管治要求

🤖 Reference:

https://www.mongodb.com/resources/basics/implementing-an-operational-data-layer
Implementing an Operational Data Layer

https://www.mongodb.com/resources/solutions/use-cases/mainframe-modernization-reference-architecture
Mainframe Modernization Reference Architecture

Editor

Danny Chan, specialty of FSI and Serverless

Kenny Chan, specialty of FSI and Machine Learning

🌐 Get started: What is MongoDB operational data layer? (Part 2) 🌐

Danny Chan — Wed, 21 Aug 2024 14:19:54 +0000

Operational Data Layer Data loading:
✅ Data must sync with source systems
✅ Appropriate data loading strategy
✅ Producer systems: frequency and quantity of data changes
✅ Consuming systems: clear requirements for data currency

Step 1: Batch extract and load:
📁 Initial batch load
📁 Copy application database data to Operational Data Layer
📁 One-time operation to load data from source systems

*Step 2: Delta extract and load: *
🔄 Starts immediately following initial batch load
🔄 Real-time synchronization
🔄 Incremental updates from source systems into the ODL
🔄 Use Change Data Capture (CDC)
🔄 Catch changes from source systems
🔍 Matching, merging, reconciling data

Data flow and maturity model:
🏗️ Simple start
🌱 Grows in scope and strategic importance
🏆 Delivering increased benefits to business

Phase 1: Simple ODL, offloading reads:
💻 Serve only read operations
💰 Cut costs
🔒 High availability: takes over during source system downtime
⚡ Improve performance
📊 Handle long-running analytics queries
📈 Handle high read traffic peak

Phase 2: Enriched ODL for new use cases:
🔍 Create single customer view
💳 Example: credit card transactions enriched by categorizing purchases
💰 Determine their spend on each category (travel)

Phase 3: Offloading reads and writes:
📥 Y-loading un both latency, new systems in parallel

Phase 4: ODL first:
✍️ All writes are directed to Operational Data Layer (ODL)

Phase 5: System of Record:
🗃️ Operational Data Layer serve as the System of Record
🏳️ Source system can be decommissioned for cost savings
🏛️ Architectural simplicity

MongoDB for Operational Data Layer:
🤗 Ease: MongoDB's document model easily manage data
🧩 Flexibility: integrate multiple source systems into a single ODL, without pre-define schema
⚡ Speed: better performance when accessing data
🎨 Versatility: satisfy a range of application requirements by flexibility of document model

Example:
📁 Embedding of arrays and sub-documents
📊 Modeling complex relationships and hierarchical data
🗄️ Ability to manipulate deeply nested data without rewrite entire document
🗂️ Model flat, table-like structures, simple key-value pairs, text
🌍 Geospatial data
🕸️ Nodes and edges used in graph processing

Processing pipelines:
🔍 Lookups and range queries
📊 Data analytics
🔨 Transformations
🔍 Faceted search
🌎 Geospatial processing
🕵️ Graph traversals

Intelligently distribute (Operational Data Layer) ODL:

Availability:
💻 Multiple copies of data using replica sets
🔄 Failover and recovery is fully automated

Scalability:
🆙 Challenge: new source systems, adding data volume, new consuming systems, increasing workload
🗄️ Large data sets
⚡ High throughput requirements
🧩 Solution - sharding:
🤖 MongoDB provides horizontal scale-out on low-cost
🔁 Automatically partitions and distributes data across multiple physical instances

Workload isolation:
🔍 Operational Data Layer able to safely serve disparate workloads
🔍 Analytical queries on up-to-date data without impact on production applications

Data locality:
🌎 Allows precise control over where data is physically stored
🗺️ Control geographic region for latency, governance requirements

Reference:

https://www.mongodb.com/resources/basics/implementing-an-operational-data-layer
Implementing an Operational Data Layer

https://www.mongodb.com/resources/solutions/use-cases/mainframe-modernization-reference-architecture
Mainframe Modernization Reference Architecture

Editor

Danny Chan, specialty of FSI and Serverless

Kenny Chan, specialty of FSI and Machine Learning

🌐 开始使用: MongoDB Operational Data Layer 是什么? (第1部分)

Danny Chan — Wed, 21 Aug 2024 14:17:14 +0000

✨ 操作数据存储(层):

与 Data Fabric、Operational Data Hub 相同
现有数据源和消费者之间的一层
无需替换遗留系统
将来自多个系统的数据合并到单一枢纽

🔍 从不同系统收集数据集

提供数据的完整图景

🔥 特设分析工具

实时分析
全面业务的分秒必争视角
不干扰操作工作负载

💡 示例:

日常职责
所需的服务级别协议 (SLA)

🏗️ 重建的基础

数字化转型的迭代方法
与遗留和新系统并行
遗留系统继续无中断运行

🚀 遗留系统现代化:

更快地构建新的业务功能
扩展到数百万用户
数据消费者访问操作数据层 (ODL)

💎 好处:

访问整个数据集
客户单一视图
人工智能流程

💻 数据即服务:

操作数据层将所有重要数据聚集在一处
应用程序和分析获得企业数据的全貌

☁️ 操作数据层的云部署:

部署在同一云提供商
与其使用系统相同的区域
渐进式、无中断的云迁移方法

🔍 挑战:

70 个不同系统分散在 15 个不同屏幕上

🔑 解决方案:

单一视图
仅使用一个屏幕访问所有信息
实时表示
360 度客户视图
产品、金融资产、业务相关实体的单一视图
来自多个源的数据

🚀 更快的客户服务时间

分析客户数据以进行交叉销售和促销

📋 用例:

客户服务代表
欺诈和风险系统
销售和营销人员
定量分析师
用户在线账户

💪 主机卸载:

单点故障
需离线维护
更轻松地将主机数据提供给新的数字渠道,而不会给遗留系统带来压力

🤔 挑战:

需要企业数据的完整视图
数据仓库或 Hadoop 基础的分析
无法满足当今对实时分析的需求
每日或每周批量加载
查询耗时长达数小时

🔍 解决方案:

针对实时决策
数据状态的分秒必争
低延迟分析查询
特设问题
示例:
- 为客户提供推荐
- 根据用户信息个性化内容
- 在企业数据上使用机器学习提取新见解
- 提高运营效率

🚀 现代化:

将现有数据暴露给新应用程序
不会对遗留系统产生潜在影响

💼 数据管理员:

从应用程序数据库获取数据,然后存储到操作数据层
使用 ETL (提取、转换、加载)、CDC (变更数据捕获)
数据必须满足要求
数据传输频率
不影响当前生产和消费应用程序

🤖 Reference:

https://www.mongodb.com/resources/basics/implementing-an-operational-data-layer
Implementing an Operational Data Layer

https://www.mongodb.com/resources/solutions/use-cases/mainframe-modernization-reference-architecture
Mainframe Modernization Reference Architecture

Editor

Danny Chan, specialty of FSI and Serverless

Kenny Chan, specialty of FSI and Machine Learning

🌐 Get started: What is MongoDB Operational Data Layer? (Part 1)

Danny Chan — Wed, 21 Aug 2024 14:14:31 +0000

✨ Operational Data Store (Layer):

Same as Data Fabric, Operational Data Hub
Layer between existing data sources and consumers
Don't need to replace legacy systems
Combine data from multiple systems into a single hub

🔍 Collect data sets from different systems

Provide a complete picture of data

🔥 Ad-hoc analytical tools

Real-time analytics
Up-to-the-minute view of the whole business
Without interfering with operational workloads

💡 Example:

Day-to-day responsibilities
Required Service Level Agreements (SLAs)

🏗️ Foundation for re-architecting

Iterative approach to digital transformation
Parallel to legacy and new systems
Legacy system continue to work without interruption

🚀 Legacy Modernization:

Build new business functions faster
Scale to millions of users
Data consumers access Operational Data Layer (ODL)

💎 Benefit:

Access entire data set
Customer single view
Artificial intelligence processes

💻 Data as a Service:

Operational Data Layer gathers all important data in one place
Applications and analytics get the full picture of enterprise data

☁️ Cloud of Operational Data Layer:

Deployed on the same cloud provider
Same regions as its consuming systems
Gradual, non-disruptive approach to cloud migration

🔍 Challenge:

70 different systems in 15 different screens

🔑 Solution:

Single view
Use only one screen to access all the information
Real-time representation
Customer 360
Single views of products, financial assets, entities relevant for business
Data from multiple sources

🚀 Faster customer call times

Analyze customer data for cross-sell and upsell opportunities

📋 Use Case:

Customer service representatives
Fraud and risk systems
Sales and marketing staff
Quantitative analysts
User online account

💪 Mainframe Offload:

Single point of failure
Taken offline for maintenance
Easier to serve mainframe data to new digital channels without straining legacy systems

🤔 Challenge:

Requires a complete view of enterprise data
Warehouse or a Hadoop-based analytics
Can't meet today's demand for real-time analytics
Loaded in daily or weekly batches
Long-running queries taking hours

🔍 Solution:

For real-time decisions
Up-to-the-minute state of data
Low latency analytics queries
Ad-hoc questions
Example:
- Recommendations for customers
- Personalizing content based on user info
- Machine learning on enterprise data to extract new insights
- Improve operational efficiency

🚀 Modernization:

Exposing existing data to new applications
Don't have potential impact to legacy systems

💼 Data Steward:

Get data from application database then store to operational data layer
Use ETL (Extract, transform, load), CDC (change data capture)
Data must fulfill requirements
Frequency of data transfer
Without affecting current producing & consuming applications

🤖 Reference:

https://www.mongodb.com/resources/basics/implementing-an-operational-data-layer
Implementing an Operational Data Layer

https://www.mongodb.com/resources/solutions/use-cases/mainframe-modernization-reference-architecture
Mainframe Modernization Reference Architecture

Editor

Danny Chan, specialty of FSI and Serverless

Kenny Chan, specialty of FSI and Machine Learning

🚀 教程：通过 MongoDB 向量搜索财务 PDF 报告 (本地嵌入)

Danny Chan — Sun, 11 Aug 2024 08:55:28 +0000

步骤 1：创建数据库集群
步骤 2：输入数据库集群信息
步骤 3：等待集群部署
步骤 4：添加网络访问白名单
步骤 5：添加数据库访问用户
步骤 6：连接到本地 Atlas 部署或 Atlas 集群
步骤 7：从 PDF 中检索文本
步骤 8：本地嵌入 PDF 文本然后插入 MongoDB
步骤 9：检查集合文档记录
步骤 10：创建向量搜索索引
步骤 11：通过向量搜索索引查询

步骤 1：创建数据库集群

步骤 2：输入数据库集群信息

步骤 3：等待集群部署

步骤 4：添加网络访问白名单

步骤 5：添加数据库访问用户

步骤 6：连接到本地 Atlas 部署或 Atlas 集群

步骤 7：从 PDF 中检索文本

步骤 8：本地嵌入 PDF 文本然后插入 MongoDB

pip install sentence-transformers==2.7.0
pip install pymongo==4.7.2
pip install langchain==0.2.6
pip install langchain-mongodb==0.1.5
pip install pandas==2.2.0
pip install langchain-openai==0.1.20
pip install langchain-chroma==0.1.0
pip install langchain-core==0.2.26
pip install langchain-huggingface==0.0.3
pip install langchain-mongodb==0.1.4

from pymongo import MongoClient
from langchain_huggingface import HuggingFaceEmbeddings

print("get documents")

data = ""
with open("./txt_final/payment.txt","r",encoding="utf8") as file:
    data = file.read()

print("Split txt into documents by page")

splits = data.split("www.iresearch.com.cn")

print("get model then embedding")

model = HuggingFaceEmbeddings(model_name="BAAI/bge-large-zh-v1.5")

print("Connect to your local Atlas deployment or Atlas Cluster")
mongo_client = MongoClient("mongodb+srv://<username>:<password>@internal-knowledge-base.xxxxx.mongodb.net/")

collection = mongo_client["internal-knowledge-base"]["papers"]

for split in splits:
    embedding = model.embed_query(split)
    collection.insert_one({ 'text_embedding': embedding, 'summary': split })

步骤 9：检查集合文档记录

PDF page 3

PDF page 4

Data structure

{
    "_id": "66b79fd22e6781dc9195820fL",
    "text_embedding": [0.019098538905382156, -0.0010181389516219497],
    "summary": "Diversified development paths for third-party payment platforms Third-party payment platforms integrate into every detail of consumer life through lightweight reach...."
}

步骤 10：创建向量搜索索引

{
  "fields": [
    {
      "type": "vector",
      "path": "text_embedding",
      "numDimensions": 1024,
      "similarity": "cosine"
    }
  ]
}

步骤 11：通过向量搜索索引查询

from pymongo import MongoClient
from langchain_huggingface import HuggingFaceEmbeddings
from langchain_mongodb import MongoDBAtlasVectorSearch
import pprint

print("Connect to your local Atlas deployment or Atlas Cluster")
mongo_client = MongoClient("mongodb+srv://<username>:<password>@internal-knowledge-base.xxxxx.mongodb.net/")

collection = mongo_client["internal-knowledge-base"]["papers"]

model = HuggingFaceEmbeddings(model_name="BAAI/bge-large-zh-v1.5")

vector_store = MongoDBAtlasVectorSearch(
   collection=collection,
   embedding=model,
   index_name="vector_index",
   embedding_key="text_embedding",
   text_key="summary"
)

query = "蚂蚁集团" # payment
results = vector_store.similarity_search(query)
pprint.pprint(results)

Result:
English version

[
    Document(metadata={'_id': {'$oid': '66b79fdc2e6781dc91958211'}}, page_content='Ant Group-Alipay Ecological Foundation}
    Document(metadata={'_id': {'$oid': '66b79fcd2e6781dc9195820e'}}, page_content='The competitive landscape of independent third-party payment platforms has formed, led by Alipay"}
    Document(metadata={'_id': {'$oid': '66b79fc32e6781dc9195820c'}}, page_content='Aikan Series Monthly Inventory of Tourism Activity in Scenic Areas"}
]

Chinese version

[
    Document(metadata={'_id': {'$oid': '66b79fdc2e6781dc91958211'}}, page_content='蚂蚁集团—支付宝生态筑基"}
    Document(metadata={'_id': {'$oid': '66b79fcd2e6781dc9195820e'}}, page_content='独立第三方支付平台竞争格局形成以支付宝为首"}
    Document(metadata={'_id': {'$oid': '66b79fc32e6781dc9195820c'}}, page_content='-艾瞰系列-景区旅游活跃度盘点月报"}
]

Reference:

https://python.langchain.com/v0.2/docs/tutorials/pdf_qa/
Build a PDF ingestion and Question/Answering system

https://www.mongodb.com/docs/atlas/atlas-vector-search/create-embeddings/
How to Create Vector Embeddings

https://www.mongodb.com/docs/atlas/atlas-vector-search/tutorials/local-rag/#std-label-local-rag
Build a Local RAG Implementation with Atlas Vector Search

https://www.mongodb.com/docs/atlas/atlas-vector-search/ai-integrations/langchain/#std-label-langchain
Get Started with the LangChain Integration

https://docs.llamaindex.ai/en/stable/examples/embeddings/huggingface/

Local Embeddings with HuggingFace

Editor

Danny Chan, specialty of FSI and Serverless

Kenny Chan, specialty of FSI and Machine Learning

🚀 Tutorial: local embedding financial PDF reports by MongoDB vector search

Danny Chan — Sun, 11 Aug 2024 00:45:40 +0000

Step 1: Create database clusters
Step 2: Input database clusters information
Step 3: Waiting clusters deploy
Step 4: Add network access whitelist
Step 5: Add database access user
Step 6: Connect to your local Atlas deployment or Atlas Cluster
Step 7: retrieve text from PDF
Step 8: local embedding PDF text then insert MongoDB
Step 9: check collections document record
Step 10: Create vector search index
Step 11: query by vector search index

Step 1: Create database clusters

Step 2: Input database clusters information

Step 3: Waiting clusters deploy

Step 4: Add network access whitelist

Step 5: Add database access user

Step 6: Connect to your local Atlas deployment or Atlas Cluster

Step 7: retrieve text from PDF

Step 8: local embedding PDF text then insert MongoDB



pip install sentence-transformers==2.7.0
pip install pymongo==4.7.2
pip install langchain==0.2.6
pip install langchain-mongodb==0.1.5
pip install pandas==2.2.0
pip install langchain-openai==0.1.20
pip install langchain-chroma==0.1.0
pip install langchain-core==0.2.26
pip install langchain-huggingface==0.0.3
pip install langchain-mongodb==0.1.4



from pymongo import MongoClient
from langchain_huggingface import HuggingFaceEmbeddings



print("get documents")

data = ""
with open("./txt_final/payment.txt","r",encoding="utf8") as file:
    data = file.read()



print("Split txt into documents by page")

splits = data.split("www.iresearch.com.cn")



print("get model then embedding")

model = HuggingFaceEmbeddings(model_name="BAAI/bge-large-zh-v1.5")



print("Connect to your local Atlas deployment or Atlas Cluster")
mongo_client = MongoClient("mongodb+srv://<username>:<password>@internal-knowledge-base.xxxxx.mongodb.net/")

collection = mongo_client["internal-knowledge-base"]["papers"]

for split in splits:
    embedding = model.embed_query(split)
    collection.insert_one({ 'text_embedding': embedding, 'summary': split })

Step 9: check collections document record

PDF page 3

PDF page 4

Data structure



{
    "_id": "66b79fd22e6781dc9195820fL",
    "text_embedding": [0.019098538905382156, -0.0010181389516219497],
    "summary": "Diversified development paths for third-party payment platforms Third-party payment platforms integrate into every detail of consumer life through lightweight reach...."
}

Step 10: Create vector search index



{
  "fields": [
    {
      "type": "vector",
      "path": "text_embedding",
      "numDimensions": 1024,
      "similarity": "cosine"
    }
  ]
}

Step 11: query by vector search index



from pymongo import MongoClient
from langchain_huggingface import HuggingFaceEmbeddings
from langchain_mongodb import MongoDBAtlasVectorSearch
import pprint



print("Connect to your local Atlas deployment or Atlas Cluster")
mongo_client = MongoClient("mongodb+srv://<username>:<password>@internal-knowledge-base.xxxxx.mongodb.net/")

collection = mongo_client["internal-knowledge-base"]["papers"]

model = HuggingFaceEmbeddings(model_name="BAAI/bge-large-zh-v1.5")

vector_store = MongoDBAtlasVectorSearch(
   collection=collection,
   embedding=model,
   index_name="vector_index",
   embedding_key="text_embedding",
   text_key="summary"
)



query = "蚂蚁集团" # payment
results = vector_store.similarity_search(query)
pprint.pprint(results)

Result:
English version



[
    Document(metadata={'_id': {'$oid': '66b79fdc2e6781dc91958211'}}, page_content='Ant Group-Alipay Ecological Foundation}
    Document(metadata={'_id': {'$oid': '66b79fcd2e6781dc9195820e'}}, page_content='The competitive landscape of independent third-party payment platforms has formed, led by Alipay"}
    Document(metadata={'_id': {'$oid': '66b79fc32e6781dc9195820c'}}, page_content='Aikan Series Monthly Inventory of Tourism Activity in Scenic Areas"}
]

Chinese version



[
    Document(metadata={'_id': {'$oid': '66b79fdc2e6781dc91958211'}}, page_content='蚂蚁集团—支付宝生态筑基"}
    Document(metadata={'_id': {'$oid': '66b79fcd2e6781dc9195820e'}}, page_content='独立第三方支付平台竞争格局形成以支付宝为首"}
    Document(metadata={'_id': {'$oid': '66b79fc32e6781dc9195820c'}}, page_content='-艾瞰系列-景区旅游活跃度盘点月报"}
]

Reference:

https://python.langchain.com/v0.2/docs/tutorials/pdf_qa/
Build a PDF ingestion and Question/Answering system

https://www.mongodb.com/docs/atlas/atlas-vector-search/create-embeddings/
How to Create Vector Embeddings

https://www.mongodb.com/docs/atlas/atlas-vector-search/tutorials/local-rag/#std-label-local-rag
Build a Local RAG Implementation with Atlas Vector Search

https://www.mongodb.com/docs/atlas/atlas-vector-search/ai-integrations/langchain/#std-label-langchain
Get Started with the LangChain Integration

https://docs.llamaindex.ai/en/stable/examples/embeddings/huggingface/

Local Embeddings with HuggingFace

Editor

Danny Chan, specialty of FSI and Serverless

Kenny Chan, specialty of FSI and Machine Learning

🌐 MongoDB在金融行业: 向量搜索和ACID事务 💰

Danny Chan — Sat, 10 Aug 2024 13:45:39 +0000

主题1: MongoDB向量搜索的应用场景

🏦 Kronos Research (台北)
💳 交易数十亿美元的加密货币
📊 分析和改进算法模型
🚀 高频加密货币交易 (HFT) 的量化研究
🤖 执行每秒大量订单的计算机程序
🌍 分析多个市场并执行订单
🔗 衍生品交易
🤖 使用大量专有市场数据训练的机器学习/AI模型
🔍 识别有利可图且可重复的市场现象
🛡️ 广泛的操作套件来控制风险并防止交易错误
🔬 即使在严重动荡的市场中也能确保正确的行为

预测智能: 🔮

🏢 尽可能靠近实际交易所的数据中心来限制延迟
🌩️ 加密货币交易所天生就在云端,允许高频交易员在物理上靠近交易所

数据格式的灵活性: 📂

🗃️ 数据并非结构化严格,如市场数据(买一/卖一价格、成交)
🤖 机器人可能有20个配置或键值对,而其他只有6个
💾 高效存储数据,分析配置如何随时间变化,以及数据如何更新和选择

Atlas数据联合: 📊

📊 图表:数据可视化,易于创建和共享
🔍 针对特定策略和仿真结果
🔍 可视化不同关系
🔍 调整交易机器人的参数

亮点: 💡

'🤔 在给定的某一天,不同配置的盈亏分布是什么样的?'

主题2: MongoDB与机器学习

MongoDB机器学习功能: 📊

💻 处理数据分析、可扩展性和分布式处理
⚡️ 通过提供实时洞见来加速洞察
🗃️ 管理从摄取到事务再到退役的数据生命周期
🚫 消除数据重复
⏱️ 针对实时处理进行优化
🔍 灵活的模型部署和模型监控(漂移检测)
🐍 集成的Python环境

MongoDB机器学习应用场景: 🚀

🚫 欺诈预防
🔧 预测性维护 - 预测和防止故障的模式
🎯 实时推荐引擎
🏭 流程优化 - 最小化成本

MongoDB中的ACID兼容事务: 💹

解决的挑战:

🔍 跨系统检索实时和归档数据,并合并结果 - 对开发人员来说是一个痛苦。
🔒 在不同方之间维护事务数据完整性,需要对多文档事务进行全有或全无的执行。

ACID兼容示例:

💳 银行 - 在账户、支付处理、交易平台之间转账,更新"系统记录"和实时仪表板。
🏥 医疗保健 - 确保患者记录准确更新且最新,防止数据异常。
🏪 库存管理 - 订单是原子性的,支付交易安全且准确,更新可用库存。

节省成本的功能: 💰
在线归档:

🗂️ 在保持数据可访问性的同时优化成本
📂 自定义规则,将不常访问的数据自动归档到云对象存储
🔍 保留通过单一端点查询归档数据的能力

Reference:

https://www.mongodb.com/products/capabilities/transactions
ACID Transactions with MongoDB

https://www.mongodb.com/blog/post/simplifying-data-science-iguazio-mongodb
IoT & IIoT — generating insights to identify patterns

https://www.mongodb.com/solutions/customer-case-studies/kronos
MongoDB Atlas Charts Enables Kronos to Trade Billions on Crypto Markets Every Day

https://www.mongodb.com/products/platform/atlas-online-archive
Online Archive. Tier your MongoDB Atlas data, query it in place.

https://www.mongodb.com/library/vector-search/vector-search-quick-start?lb-mode=overlay
Atlas Vector Search Quick Start

https://www.mongodb.com/developer/products/atlas/agent-fireworksai-mongodb-langchain/
Building an AI Agent With Memory Using MongoDB, Fireworks AI, and LangChain

https://www.mongodb.com/developer/products/mongodb/langchain-vector-search/
Introduction to LangChain and MongoDB Atlas Vector Search

Editor

Danny Chan, specialty of FSI and Serverless

Kenny Chan, specialty of FSI and Machine Learning

🌐 MongoDB in the Financial Industry: Vector Search and ACID-Compliant Transactions 💰

Danny Chan — Sat, 10 Aug 2024 13:43:53 +0000

Topic 1: MongoDB Vector Search Use Cases

- 🏦 Kronos Research (Taipei)

💳 Trades billions of dollars in cryptocurrency
📊 Analyzes and improves algorithmic models
🚀 Quantitative research for high-frequency cryptocurrency trading (HFT)
🤖 Computer programs to transact high volumes of orders in seconds
🌍 Analyzes multiple markets and executes orders
🔗 Derivatives trading
🤖 Machine learning/AI models trained on large volumes of proprietary market data
🔍 Identifies profitable and repeatable market phenomena
🛡️ Extensive operations suite to control risk and prevent trading errors
🔬 Ensures correct behavior even during severe market turbulence

Prediction Intelligence: 🔮

🏢 Data centers as close as possible to actual exchanges to limit latency
🌩️ Crypto exchanges are natively in the cloud, allowing high-frequency traders to be physically located close to the exchanges

Data Format Flexibility: 📂

🗃️ Data are not structurally rigid, like market data (bid and ask prices, trades)
🤖 Bots might have 20 configurations or key-value pairs, while others have only 6
💾 Efficiently store data and analyze how configurations change over time, and how data is updated and selected

Atlas Data Federation: 📊

📊 Charts: Data visualization, easy to create and share
🔍 For specific strategies and simulation results
🔍 Visualize the different relationships
🔍 Adjust the dials for trading bots

Highlight: 💡

'🤔 On a given day, what's the distribution of profit and loss results across the different configurations?'

Topic 2: MongoDB and Machine Learning

MongoDB Machine Learning Capabilities: 📊

💻 Handles data analytics, scalability, and distributed processing
⚡️ Accelerates insights by delivering real-time intelligence
🗃️ Manages the data lifecycle from ingestion to transactions to retirement
🚫 Eliminates data duplication
⏱️ Optimized for real-time processing
🔍 Flexible model deployment and model monitoring (drift detection)
🐍 Integrated Python environment

MongoDB Machine Learning Use Cases: 🚀

🚫 Fraud prevention
🔧 Predictive maintenance - patterns to predict and prevent failures
🎯 Real-time recommendation engines
🏭 Process optimization - minimizing costs

ACID-Compliant Transactions in MongoDB: 💹
Challenges Solved:

🔍 Separate queries to retrieve live and archival data across systems, and merging the results - a pain for developers.
🔒 Maintaining transactional data integrity between different parties, requiring all-or-nothing execution for multi-document transactions.

ACID-Compliant Examples:

💳 Bank - Transfer of funds between accounts, payment processing, trading platforms, updating the "System of Record" and real-time dashboards.
🏥 Healthcare - Ensuring patient records are updated accurately and up-to-date, preventing data anomalies.
🏪 Inventory Management - Orders are atomic, payment transactions are secure and accurate, updating available inventory.

Cost-Saving Feature: 💰
Online Archive:

🗂️ Optimize costs while keeping data accessible
📂 Custom rules to automatically archive infrequently accessed data to cloud object storage
🔍 Retain the ability to query archived data through a single endpoint

Reference:

https://www.mongodb.com/products/capabilities/transactions
ACID Transactions with MongoDB

https://www.mongodb.com/blog/post/simplifying-data-science-iguazio-mongodb
IoT & IIoT — generating insights to identify patterns

https://www.mongodb.com/solutions/customer-case-studies/kronos
MongoDB Atlas Charts Enables Kronos to Trade Billions on Crypto Markets Every Day

https://www.mongodb.com/products/platform/atlas-online-archive
Online Archive. Tier your MongoDB Atlas data, query it in place.

https://www.mongodb.com/library/vector-search/vector-search-quick-start?lb-mode=overlay
Atlas Vector Search Quick Start

https://www.mongodb.com/developer/products/atlas/agent-fireworksai-mongodb-langchain/
Building an AI Agent With Memory Using MongoDB, Fireworks AI, and LangChain

https://www.mongodb.com/developer/products/mongodb/langchain-vector-search/
Introduction to LangChain and MongoDB Atlas Vector Search

Editor

Danny Chan, specialty of FSI and Serverless

Kenny Chan, specialty of FSI and Machine Learning

🔒 应用技巧: MongoDB安全、合规性与零信任 🔑

Danny Chan — Sat, 10 Aug 2024 13:41:08 +0000

主题1: 监管合规性 💼

🚨 检测欺诈、风险管理和客户身份验证 (KYC)
🛡️ 制裁机制 (Sanctions) 和反洗钱 (AML) 对虚拟货币和预付卡的应用
🔍 利用AI进行数据清洗和监控系统,以识别可疑活动

欺诈预防: 🚫

🔍 提高检测精度
🌐 提高灵活性和可扩展性
🔒 提高安全性
💰 降低运营成本

实时预测异常: 🕰️

🧭 建立客户全面画像
🚨 更好地预测交易异常

保护数据: 🔒

📜 确保合规性
🔐 行业领先的加密、访问控制和数据保护协议

变更流: 🔄

👀 实时监控数据库变更

操作数据存储 (ODS): 🗃️

💾 处理大量数据
🚀 实时捕获、存储和处理高吞吐量的交易数据

监管合规性: 📜

💳 支付卡行业数据安全标准 (PCI)
🌍 通用数据保护条例 (GDPR)
🇺🇸 加州消费者隐私法 (CCPA)
💳 支付服务指令 2 (PSD2)
🌳 可持续金融披露条例 (SFDR)
🔑 基于角色的授权控制
🗺️ 地理空间和网络图分析
🌍 识别与气候变化相关的物理风险 (如洪水、山火)
🔮 采用预测性方法理解复杂模式

🔒 主题2: 零信任 🔑

零信任: 🕸️

🔒 网络安全方法
🔐 限制用户可能损害安全性的机会
🚫 限制互联网连接
👤 为所有用户实施强身份验证
🔒 严格限制对数据的访问
🔐 加密数据
🔑 所有操作都必须通过用户授权
✅ 允许用户自行验证

默认安全: 🔒

🌐 配置IP访问列表
🔑 允许对数据库进行身份验证尝试

审计功能: 🔍

👀 跟踪事件、用户或角色的可疑或意外行为

Reference:

https://www.mongodb.com/solutions/industries/financial-services/fraud-prevention
Fraud Prevention with MongoDB

https://www.mongodb.com/library/financial-services/powering-innovation-fin-serv-ai?lb-mode=overlay
Powering Innovation in Financial Services with Artificial Intelligence

MongoDB: Capabilities for Use in a Zero Trust Environment
https://www.mongodb.com/library/financial-services/wp-mongodb-capabilities-for-use-zero-trust-environment?lb-mode=overlay

Editor

Danny Chan, specialty of FSI and Serverless

Kenny Chan, specialty of FSI and Machine Learning

🔒 Tip and Trick: MongoDB Security, Regulatory Compliance & Zero Trust 🔑

Danny Chan — Sat, 10 Aug 2024 13:39:20 +0000

Topic 1: Regulatory Compliance 💼

🚨 Detecting Fraud, Risk Management, and Customer Identity Verification (KYC)
🛡️ Sanctions Regimes (Sanctions) and Anti-Money Laundering (AML) for Virtual Currencies & Prepaid Cards
🔍 AI-powered Data Cleansing and Surveillance Systems to Identify Suspicious Activities

Fraud Prevention: 🚫

🔍 Improves Detection Accuracy
🌐 Increases Flexibility and Scalability
🔒 Enhances Security
💰 Reduces Cost of Operation

Predict Anomalies in Real-Time: 🕰️

🧭 Create a Holistic Picture of Customers
🚨 Better Predict Transaction Anomalies

Protect Data: 🔒

📜 Ensure Regulatory Compliance
🔐 Industry-Leading Encryption, Access Controls, and Data Protection Protocols

Change Streams: 🔄

👀 Monitoring Database Changes in Real-Time

Operational Data Store (ODS): 🗃️

💾 Handle Large Amounts of Data
🚀 Capture, Store, and Process High-Volume Transactional Data in Real-Time

Regulatory Compliance: 📜

💳 Payment Card Industry Data Security Standard (PCI)
🌍 General Data Protection Regulation (GDPR)
🔑 California Consumer Privacy Act (CCPA)
💳 Payment Services Directive 2 (PSD2)
🌳 Sustainable Finance Disclosure Regulation (SFDR)
🔑 Role-Based Authorization Controls
🗺️ Geospatial & Network Graph Analytics
🌍 Identify Physical Risks Associated with Climate Change (e.g., Floods, Wildfires)
🔮 Predictive Approach to Understand Intricate Patterns

🔒 Topic 2: Zero Trust 🔑

Zero Trust: 🕸️

🔒 Cybersecurity Approach
🔐 Limit Users' Opportunities to Compromise Security
🚫 Limited Connectivity to the Internet
👤 Robust Authentication for All Users
🔒 Strictly Limit Access to Data
🔐 Encrypt Data
🔑 All Actions Must Pass Through User Authorizations
✅ Allow Users to Authenticate Themselves

Security by Default: 🔒

🌐 Configure IP Access Lists
🔑 Allow Attempts to Authenticate to Database

Auditing Capabilities: 🔍

👀 Track Suspicious or Unexpected Behavior by Event, User, or Role

Reference:

https://www.mongodb.com/solutions/industries/financial-services/fraud-prevention
Fraud Prevention with MongoDB

https://www.mongodb.com/library/financial-services/powering-innovation-fin-serv-ai?lb-mode=overlay
Powering Innovation in Financial Services with Artificial Intelligence

MongoDB: Capabilities for Use in a Zero Trust Environment
https://www.mongodb.com/library/financial-services/wp-mongodb-capabilities-for-use-zero-trust-environment?lb-mode=overlay

Editor

Danny Chan, specialty of FSI and Serverless

Kenny Chan, specialty of FSI and Machine Learning

🚀 行业洞见: MongoDB 推动支付业务 💸

Danny Chan — Wed, 07 Aug 2024 15:02:16 +0000

金融领域的 AI 挑战:

🤥 误导性建议
🔍 错误的风险评估
🔒 敏感信息
🚫 幻觉缓解 (由检索增强生成 - RAG 解决)
📊 数据质量、相关性和准确性

非卡支付中的 AI:

📚 交易记录的历史"库存"
🌊 实时"流"数据

现金流分析:

🔮 预测: 实时洞见现金状况
💰 营运资本优化: 流动性管理, 向企业财务主管提供建议, 增值
🔍 支付处理优化: 自动化工作流程
🚨 风险和欺诈: 交易筛查, 避免手动干预

数据驱动的增值服务:

🧠 为企业财务主管和高级财务主管量身定制服务和解决方案
🌍 根据不同行业和地区的需求定制产品

实时数据可见性和预测:

🔍 提供实时现金头寸可见性
📊 提供跨机构仪表板以获得全面的数据洞察

增值数据洞见:

🔮 提供基于情景的预测能力
🤖 根据数据分析提供建议性措施
💯 生成未来头寸的风险评分以帮助决策

改善支付服务:

🤖 自动化付款跟踪和对账流程
⚡️ 提高付款相关服务的效率和准确性

中央仪表板:

💻 从多家银行实时整合数据到单一仪表板
💰 实时现金预测和余额
🛡️ 欺诈保护
📊 定制内部报告
💳 应收账款对账
📄 符合 SO20022 标准
🔍 数据分析 (头寸风险评分)

带地图数据的中央仪表板:

🌍 纬度和经度坐标
📍 地理区域(国家、州)名称文本
🗺️ 色块图: 预定义的地理区域形状
📍 散点图: 指示数据点的标记
🔥 热力图: 数据点颜色强度

MongoDB 的更多优势:

🔍 二进制 JSON 格式 (BSON): 更快的解析, 可搜索和索引
⚡️ 对ad-hoc查询的性能不断提高
🔍 在大规模下对字段查询、范围查询和正则表达式搜索有显著差异
💾 时间点恢复: 采用 MongoDB 的操作日志 (oplog), 持续备份, 一致快照

Editor

Danny Chan, specialty of FSI and Serverless

Kenny Chan, specialty of FSI and Machine Learning

🚀 Industry Insights: MongoDB Boosts Payment Business 💸

Danny Chan — Wed, 07 Aug 2024 14:59:03 +0000

AI Challenges in Finance:

🤥 Misleading advice
🔍 Incorrect risk assessments
🔒 Sensitive information
🚫 Hallucination mitigation (solved by Retrieval Augmented Generation - RAG)
📊 Data quality, relevance, and accuracy

AI in Non-Card Payments:

📚 Historic "stock" of transaction records
🌊 "Flow" in the form of live data

Cash Flow Analysis:

🔮 Forecasting: real-time insights into cash position
💰 Working capital optimization: liquidity management, recommendations to corporate treasurers, value-adding
🔍 Payment processing optimization: automate workflow
🚨 Risk and fraud: transaction screening, avoid manual intervention

Data-Led Value-Added Services:

🧠 Customized services and solutions for corporate treasurers and senior finance executives
🌍 Tailored offerings based on business verticals and regions

Real-Time Data Visibility and Forecasts:

🔍 Offer real-time visibility into cash positions
📊 Provide cross-institution dashboards for comprehensive data insights

Value-Adding Data Insights:

🔮 Deliver scenario-based forecasting capabilities
🤖 Offer recommended actions based on data analysis
💯 Generate risk scores on future positions to aid decision-making

Improved Payment Services:

🤖 Automate payment tracking and reconciliation processes
⚡️ Enhance efficiency and accuracy of payment-related services

Central Dashboard:

💻 Real-time consolidated data from multiple banks into a single dashboard
💰 Real-time cash forecasting and balances
🛡️ Fraud protection
📊 Custom reporting for internal use
💳 Receivables reconciliation
📄 SO20022 compliance
🔍 Data analytics (risk scores on positions)

Central Dashboard with Map Data:

🌍 Latitude and longitude coordinates
📍 Text with names of geographical areas (countries, states)
🗺️ Choropleth: predefined shapes for geographical areas
📍 Scatter: data markers to indicate data points
🔥 Heatmap: color intensity of data points

More MongoDB Advantages:

🔍 Binary JSON format (BSON): faster parsing, searched and indexed
⚡️ Increasing performance for ad-hoc queries
🔍 Significant difference at scale for field queries, range queries, regular expression searches
💾 Point-in-time recovery: adopt MongoDB's operation log (oplog), continuous backup, consistent snapshots

Editor

Danny Chan, specialty of FSI and Serverless

Kenny Chan, specialty of FSI and Machine Learning