Skip to content

Commit 9740d54

Browse files
wasin-waeosriwasin-waeosri
committed
Add LDPv2 configuration and remove deprecated app.py script, then add the README file
1 parent 5ad6153 commit 9740d54

4 files changed

Lines changed: 224 additions & 9 deletions

File tree

‎README.md‎

Lines changed: 191 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,191 @@
1+
# Machine Readable News Example with LSEG Data Library for Python
2+
3+
- Last update: January 2025
4+
- Compiler: Python
5+
- Prerequisite: The Real-Time Distribution System or The Real-Time -- Optimized credentials (V1) with MRN service
6+
7+
This example shows how developers may use the [LSEG Data Library for Python](https://developers.lseg.com/en/api-catalog/lseg-data-platform/lseg-data-library-for-python) Delivery layer feature to subscribe to the Machine Readable News (MRN) from LSEG Real-Time Distribution System (RTDS) and LSEG Real-Time Optimized (RTO). The example just connects to LSEG Real-Time platform via a WebSocket connection, then subscribes and displays MRN News data in a console.
8+
9+
**Note:** The news message is in UTF-8 JSON string format. Some news messages that contain special Unicode character may not be able to show in Windows OS console (cmd, git bash, powershell, etc) due to the OS limitation. Those messages will be print as ```UnicodeEncodeError exception. Cannot decode Unicode character``` message in a console instead.
10+
11+
## <a id="prerequisite"></a> Prerequisite
12+
13+
This example requires the following dependencies.
14+
15+
1. The Real-Time Distribution System (RTDS) or Real-Time Optimized (RTO) with MRN Service.
16+
2. [Python](https://www.python.org/) interpreter and runtime.
17+
3. RTO Access credentials (Version 1 or Version 2) for the RTO example.
18+
4. Internet connection.
19+
20+
Please contact your LSEG representative to help you to access the RTO account and services.
21+
22+
## <a id="news_mrn_overview"></a>Machine Readable News Overview
23+
24+
Machine Readable News (MRN) is an advanced service for automating the consumption and systematic analysis of news. It delivers deep historical news archives, ultra-low latency structured news and news analytics directly to your applications. This enables algorithms to exploit the power of news to seize opportunities, capitalize on market inefficiencies, and manage event risk.
25+
26+
### <a id="mrn_data_model"></a>MRN Data model
27+
28+
MRN is published over the Real-Time platform using an Open Message Model (OMM) envelope in News Text Analytics domain messages. The Real-time News content set is made available over MRN_STORY RIC. The content data is contained in a FRAGMENT field that has been compressed and potentially fragmented across multiple messages, to reduce bandwidth and message size.
29+
30+
A FRAGMENT field has a different data type based on a connection type:
31+
32+
- RSSL connection (RTSDK [C++](https://developers.lseg.com/en/api-catalog/refinitiv-real-time-opnsrc/rt-sdk-cc)/[Java](https://developers.lseg.com/en/api-catalog/refinitiv-real-time-opnsrc/rt-sdk-java)/[C#](https://developers.lseg.com/en/api-catalog/refinitiv-real-time-opnsrc/rt-sdk-cc)): BUFFER type
33+
- WebSocket connection: Base64 ASCII string
34+
35+
The data goes through the following series of transformations:
36+
37+
1. The core content data is a UTF-8 JSON string
38+
2. This JSON string is compressed using gzip
39+
3. The compressed JSON is split into several fragments (BUFFER or Base64 ASCII string) which each fit into a single update message
40+
4. The data fragments are added to an update message as the FRAGMENT field value in a FieldList envelope
41+
42+
![Figure-1](images/mrn_process.png "MRN data compression process")
43+
44+
Therefore, to parse the core content data, the application will need to reverse this process. The WebSocket application also needs to convert a received Base64 string in a FRAGMENT field to bytes data before further process this field. This application uses Python [base64](https://docs.python.org/3/library/base64.html) and [zlib](https://docs.python.org/3/library/zlib.html) modules to decode Base64 string and decompress JSON string.
45+
46+
If you are not familiar with MRN concept, please visit the following resources which will give you a full explanation of the MRN data model and implementation logic:
47+
48+
- [Webinar Recording: Introduction to Machine Readable News](https://developers.lseg.com/news#news-accordion-nid-12045)
49+
- [Introduction to Machine Readable News (MRN) with Enterprise Message API (EMA)](https://developers.lseg.com/en/article-catalog/article/introduction-machine-readable-news-mrn-elektron-message-api-ema).
50+
- [MRN Data Models and the Real-Time SDK Implementation Guide](https://developers.lseg.com/en/api-catalog/refinitiv-real-time-opnsrc/rt-sdk-java/documentation#mrn-data-models-implementation-guide).
51+
- [Introduction to Machine Readable News with WebSocket API](https://developers.lseg.com/en/article-catalog/article/introduction-machine-readable-news-elektron-websocket-api-refinitiv).
52+
53+
54+
## <a id="how_to_run"></a>How to run this example
55+
56+
The first step is to unzip or download the example project folder into a directory of your choice.
57+
58+
It is an advisable to create a dedicate Python environment to run each Python project. You can create a new [Python virtual environment](https://docs.python.org/3/library/venv.html) to run this example with the following steps:
59+
60+
1. Open the ```src/lseg_data.config.json``` file and add your RTO or RTDS information based on your preference
61+
62+
```json
63+
"sessions": {
64+
"default": "platform.ldp",
65+
"platform": {
66+
"ldp": {
67+
"app-key": "YOUR APP KEY GOES HERE!",
68+
"username": "YOUR LDP LOGIN OR MACHINE GOES HERE!",
69+
"password": "YOUR LDP PASSWORD GOES HERE!",
70+
"signon_control":true
71+
},
72+
"ldpv2":{
73+
"client_id": "Service-ID (Client ID V2)",
74+
"client_secret": "Client Secret",
75+
"signon_control":true
76+
},
77+
"deployed": {
78+
"app-key": "YOUR APP KEY GOES HERE!",
79+
"realtime-distribution-system": {
80+
"url" : "YOUR DEPLOYED HOST:PORT GOES HERE!",
81+
"dacs" : {
82+
"username" : "YOUR DACS ID GOES HERE!",
83+
"application-id" : 256,
84+
"position" : ""
85+
}
86+
}
87+
}
88+
}
89+
}
90+
```
91+
2. Please note that the ```platform``` configuration is based on your connection information:
92+
93+
- if you are using the RTO with the *Version 1 Authentication*, the ```default``` value must be ```platform.ldp```.
94+
- if you are using the RTO with the *Version 2 Authentication*, the ```default``` value must be ```platform.ldpv2```.
95+
- if you are using the *local RTDS connection*, the ```default``` value must be ```platform.deployed```.
96+
97+
3. Open a command prompt application and go to the project folder
98+
4. Run the following command in a command prompt to create a new Python virtual environment named *venv* for the project (the name can be changed based on your preference).
99+
100+
```bash
101+
$ python -m venv venv
102+
```
103+
5. Once the environment creation is succeed, activate a virtual environment with the following command.
104+
105+
```bash
106+
#Windows
107+
$ venv\Scripts\activate
108+
109+
#Linux
110+
$ source venv/bin/activate
111+
112+
#success
113+
(venv)$
114+
```
115+
6. Run the following command to install the dependencies in virtual environment.
116+
117+
```bash
118+
(venv)$ pip install -r requirements.txt
119+
```
120+
121+
7. To run an application go to the ```src``` folder while the ```venv``` environment is activated, and run the following command
122+
123+
```bash
124+
(venv)$src> python OMMStream-MRN-LD.py
125+
```
126+
127+
8. Press ```Ctrl+C``` buttons to stop the application, and run the following command to deactivate the Python virtual environment.
128+
129+
```bash
130+
(venv)$src> deactivate
131+
132+
$src>
133+
```
134+
### Note about the Virtual Environment
135+
136+
If you are using the [Anaconda](https://anaconda.org/anaconda/conda)/[Miniconda](https://docs.anaconda.com/miniconda/) Python distribution, please refer to the [Conda document](https://docs.conda.io/projects/conda/en/latest/user-guide/getting-started.html) about how to create a virtual environment using Conda.
137+
138+
## Example Results
139+
140+
### MRN Data Refresh Message
141+
142+
```json
143+
>>> Refresh event received at 14:05:40.076276
144+
{
145+
"ID": 5,
146+
"Type": "Update",
147+
"Domain": "NewsTextAnalytics",
148+
"UpdateType": "Unspecified",
149+
"DoNotConflate": true,
150+
"DoNotRipple": true,
151+
"DoNotCache": true,
152+
"Key": {
153+
"Service": "ELEKTRON_DD",
154+
"Name": "MRN_STORY"
155+
},
156+
"PermData": "AwEBEBU8",
157+
"SeqNumber": 54958,
158+
"Fields": {
159+
"TIMACT_MS": 26035856,
160+
"ACTIV_DATE": "2025-01-06",
161+
"MRN_TYPE": "STORY",
162+
"MRN_V_MAJ": "2",
163+
"MRN_V_MIN": "10",
164+
"TOT_SIZE": 716,
165+
"FRAG_NUM": 1,
166+
"GUID": "Pt3BVKYG__2501062wgGRnlW/ulmGvzHnd35kqZ6qc0/chruUU4ph2",
167+
"MRN_SRC": "HK1_PRD_A",
168+
"FRAGMENT": "H4sIAAAAAAAC/8VUUW/TMBB+51ec8gRS0ybp2iG/tdlWCmsa2oxpY2hyE6cxuHbmOB0B8d85J+0mISbxxov93dm++8732T8dKsw8c4gjYzOcfvpwM3N6Dq0zzmTKKod8dqKYxNML9HYgjJwvPWejsgYPeeP3VAajAXiBP4TJAs7XycA7RTxbJHfSdY+YQCyoMRVM4kkIU64wQcUExEVT8ZQKAknBXlqEUiskUwGvIKcpF9xQwzIwhVb1tsCZAbvmMlOP8DouaMVgPn8DhqWFVEJtmz6EqpaG6ZJq00CqWcaNjaZKJiFXGqgQkKpdSSWmxoDUANUMpDJgz/CUl9RwuQUu23RHRhnXLDWieeJy4NGHeQ6NqtsoVb3ZcdMeR5fGGJhyh/GUfDpHJVhSlorMoKD7LjkVmtGssZjnHGs+XKPKcR9W8peyNKuM5qmNXvWgFMzeB/Ld84zBo0YiWHMXL+04YLF2lwElGRSWYak5MjGqrbUyGN5mtMZiGT6342USORfoqkAy2ycFG+yQpBuBFgbeYYdtNX04Jf6QjEb98TA4SuZQIcogr1EEE8G0cd07iQrMua5MiDeC3Uf1BV4wcj3f9caJdwx0OgpucWeBtya4ZLgrvpwkyZqA77qY7r/pEElxS/r4zO7vg5Hne+PgcTtbSXE9qMVutv/xTmbD0beH2/FD6g3SQtdXVydlEdjTEhthH+Uyx2eJb1BQua3p1tbI7O3skBqaSVOiK0Cb7w6GY9h3MygF5XbfQQsa/dGaxJfJ2jrrzRqrqfHJO5jHkLqy7cIVVO9X1Hj7F0zICXoWxP941s5BmOB8Rd62Y9iOi3aMcFzZ/8LzPP8ZP8Ew8v6wrBEFZDJbzTs0nS8vOhSeJTcHtFx04PI86kC0mnVgFV3bn8nQb2zNHmr7fznE7zm13iLG32rYc/aoSVT8P0jo16vf3flwkBsFAAA="
169+
}
170+
}
171+
FRAGMENT length = 716
172+
decompress News FRAGMENT(s) for GUID Pt3BVKYG__2501062wgGRnlW/ulmGvzHnd35kqZ6qc0/chruUU4ph2
173+
News = {'altId': 'nPt3BVKYG', 'audiences': ['NP:PBF', 'NP:PBFCN'], 'body': '06Jan25/ 0213 AM EST/0713 GMT\n--0713 GMT: Platts APAC Biodiesel Physical: The APAC Biodiesel Physical process is facilitated through the eWindow (Phase II) technology. Counterparty credit is open for all companies that are not participating in the process directly through eWindow. If you are submitting your information through an editor and have not already notified Platts of any counterparty credit restrictions, please provide written notification at least one hour prior to the start of the MOC process if any counterparty credit filters need to be enabled or modified. 7:13:55.632 GMT\n--Platts Biofuel Alert--\n', 'firstCreated': '2025-01-06T07:13:55.752Z', 'headline': 'PLATTS: 1-- 7: Platts APAC Biodiesel Physical: The APAC Biodiesel Physical process is facilitated through the eWindow (Phase II) techno', 'id': 'Pt3BVKYG__2501062wgGRnlW/ulmGvzHnd35kqZ6qc0/chruUU4ph2', 'instancesOf': [], 'language': 'en', 'messageType': 2, 'mimeType': 'text/plain', 'provider': 'NS:PLTS', 'pubStatus': 'stat:usable', 'subjects': ['A:4', 'M:1QD', 'M:2CT', 'U:8', 'U:C', 'U:M', 'U:N', 'R:PBF0001', 'R:PBF001', 'R:PBFCN0001', 'R:PBFCN001', 'N2:AGRI', 'N2:BIOF', 'N2:CDTY', 'N2:COM', 'N2:LEN', 'N2:NRG', 'N2:RNW'], 'takeSequence': 1, 'urgency': 3, 'versionCreated': '2025-01-06T07:13:55.752Z'}
174+
```
175+
176+
## <a id="references"></a>References
177+
178+
For further details, please check out the following resources:
179+
180+
- [LSEG Data Library for Python](https://developers.lseg.com/en/api-catalog/lseg-data-platform/lseg-data-library-for-python)
181+
- [Introduction to Machine Readable News with WebSocket API](https://developers.lseg.com/en/article-catalog/article/introduction-machine-readable-news-elektron-websocket-api-refinitiv).
182+
- [WebSocket API Machine Readable News Example with Python](https://github.com/LSEG-API-Samples/Example.WebSocketAPI.Python.MRN) on GitHub.
183+
- [Machine Readable News (MRN) & N2_UBMS Comparison and Migration Guide](https://developers.lseg.com/en/article-catalog/article/machine-readable-news-mrn-n2_ubms-comparison-and-migration-guide).
184+
- [MRN WebSocket JavaScript example on GitHub](https://github.com/LSEG-API-Samples/Example.WebSocketAPI.Javascript.NewsMonitor).
185+
- [MRN WebSocket C# NewsViewer example on GitHub](https://github.com/LSEG-API-Samples/Example.WebSocketAPI.CSharp.MRNWebSocketViewer).
186+
- [Developer Article: Introduction to Machine Readable News with WebSocket API](https://developers.lseg.com/en/article-catalog/article/introduction-machine-readable-news-elektron-websocket-api-refinitiv).
187+
- [Account authorization V1 to V2 migration cheat sheet](https://developers.lseg.com/en/article-catalog/article/account-authorization-v1-to-v2-migration-cheat-sheet) article.
188+
- [Getting Started with Version 2 Authentication for Real-Time - Optimized: Overview](https://developers.lseg.com/en/article-catalog/article/getting-started-with-version-2-authentication-for-refinitiv-real) article.
189+
190+
191+
For any questions related to this example or the LSEG Data Library, please use the Developer Community [Q&A Forum](https://community.developers.refinitiv.com/).

‎images/mrn_process.png‎

62.1 KB
Loading

src/app.py renamed to src/OMMStream-MRN-LD.py

Lines changed: 16 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@
77

88

99
#!/usr/bin/env python
10+
import sys
1011
import lseg.data as ld
1112
from lseg.data.delivery import omm_stream
1213
import datetime
@@ -20,6 +21,11 @@
2021
_news_envelopes = []
2122
RIC_CODE = 'MRN_STORY'
2223
DOMAIN = 'NewsTextAnalytics'
24+
SERVICE = 'ELEKTRON_DD'
25+
26+
# Config the encoding for the console
27+
sys.stdin.reconfigure(encoding='utf-8')
28+
sys.stdout.reconfigure(encoding='utf-8')
2329

2430
# Retrieve data
2531
# Callback function to display data or status events
@@ -118,8 +124,9 @@ def process_mrn_update(message_json):
118124

119125
# Create an OMM stream and register event callbacks
120126
stream = omm_stream.Definition(
121-
name=RIC_CODE,
122-
domain= DOMAIN).get_stream()
127+
name = RIC_CODE,
128+
domain = DOMAIN,
129+
service = SERVICE).get_stream()
123130

124131
# Define the event callbacks
125132
# Refresh - the first full image we get back from the server
@@ -141,10 +148,10 @@ def process_mrn_update(message_json):
141148
# We should receive the initial Refresh for the current field values
142149
# followed by updates for the fields as and when they occur
143150

144-
try:
145-
while True:
146-
time.sleep(1)
147-
except KeyboardInterrupt:
148-
stream.close()
149-
# Close the session
150-
ld.close_session()
151+
try:
152+
while True:
153+
time.sleep(1)
154+
except KeyboardInterrupt:
155+
stream.close()
156+
# Close the session
157+
ld.close_session()

‎src/lseg-data.config.json‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,23 @@
1919
"username": "YOUR LDP LOGIN OR MACHINE GOES HERE!",
2020
"password": "YOUR LDP PASSWORD GOES HERE!",
2121
"signon_control":true
22+
},
23+
"ldpv2":{
24+
"client_id": "Service-ID (Client ID V2)",
25+
"client_secret": "Client Secret",
26+
"signon_control":true,
27+
"app-key": ""
28+
},
29+
"deployed": {
30+
"app-key": "YOUR APP KEY GOES HERE!",
31+
"realtime-distribution-system": {
32+
"url" : "YOUR DEPLOYED HOST:PORT GOES HERE!",
33+
"dacs" : {
34+
"username" : "YOUR DACS ID GOES HERE!",
35+
"application-id" : 256,
36+
"position" : ""
37+
}
38+
}
2239
}
2340
}
2441
}

0 commit comments

Comments
 (0)