NEW ENTRIES
Basics data process
15/3/2021
Study notes
Types of data
- Structured
table-based source systems such as a relational database or from a flat file such as a comma separated (CSV) file
The primary element of a structured file is that the rows and columns are aligned consistently throughout the file. - Semi-structured
data such as JavaScript object notation (JSON) files, which may require flattening prior to loading into your source system.
When flattened, this data doesn't have to fit neatly into a table structure. - Unstructured
data stored as key-value pairs that don't adhere to standard relational models and Other types of unstructured data that are commonly used include portable data format (PDF), word processor documents, and images.
- Operational data
Usually transactional data that is generated and stored by applications, often in a relational or non-relational database. - Analytical data
Data that has been optimized for analysis and reporting, often in a data warehouse. - Streaming data
Perpetual sources of data that generate data values in real-time, often relating to specific events.
Common sources of streaming data include internet-of-things (IoT) devices and social media feeds. - Data pipelines
Used to orchestrate activities that transfer and transform data.
Pipelines are the primary way in which data engineers implement repeatable extract, transform, and load (ETL) solutions that can be triggered based on a schedule or in response to events. - Data lakes
Storage repository that holds large amounts of data in native, raw formats
Data lake stores are optimized for scaling to massive volumes (terabytes or petabytes) of data. - Data warehouses
Centralized repository of integrated data from one or more disparate sources.
Data warehouses store current and historical data in relational tables that are organized into a schema that optimizes performance for analytical queries. - Apache Spark
Parallel processing framework that takes advantage of in-memory processing and a distributed file storage. It's a common open-source software (OSS) tool for big data scenarios.
- Data integration
Establishing links between operational and analytical services and data sources to enable secure, reliable access to data across multiple systems. - Data transformation
Operational data usually needs to be transformed into suitable structure and format for analysis
It is often as part of an extract, transform, and load (ETL) process; though increasingly a variation in which you extract, load, and transform (ELT) the data is used to quickly ingest the data into a data lake and then apply "big data" processing techniques to transform it. Regardless of the approach used, the data is prepared to support downstream analytical needs. - Data consolidation
Combining data that has been extracted from multiple data sources into a consistent structure - usually to support analytics and reporting.
Commonly, data from operational systems is extracted, transformed, and loaded into analytical stores such as a data lake or data warehouse.
- Operational data is generated by applications and devices and..
- Stored in Azure data storage services such as Azure SQL Database, Azure Cosmos DB, and Microsoft Dataverse.
- Streaming data is captured in event broker services such as Azure Event Hubs.
- Operational data must be captured, ingested, and consolidated into analytical store and ...
- From where it can be modeled and visualized in reports and dashboards.
The core Azure technologies used to implement data engineering workloadsinclude:
- Azure Synapse Analytics
Azure Synapse Analyticsincludes functionality for pipelines, data lakes, and relational data warehouses. - Azure Data Lake Storage Gen2
- Azure Stream Analytics
- Azure Data Factory
- Azure Databricks
Azure Data Lake Storage Gen2
Provides a cloud-based solution for data lake storage in Microsoft Azure, and underpins many large-scale analytics solutions built on Azure.
A data lake is a repository of data that is stored in its natural format, usually as blobs or files. Azure Data Lake Storage is a comprehensive, massively scalable, secure, and cost-effective data lake solution for high performance analytics built into Azure.
- Hadoop compatible access.
You can store the data in one place and access it through compute technologies including Azure Databricks, Azure HDInsight, and Azure Synapse Analytics - Security
Data Lake Storage supports access control lists (ACLs) and Portable Operating System Interface (POSIX) permissions that don't inherit the permissions of the parent directory - Performance
- Data redundancy
- Blob
in terms of blob manageability the blobs are stored as a single-level hierarchy in a flat namespace.
Flat namespaces, by contrast, require several operations proportionate to the number of objects in the structure. - Azure Data Lake Storage Gen2
builds on blob storage and optimizes I/O of high-volume data by using a hierarchical namespace that organizes blob data into directories, and stores metadata about each directory and the files within it.
Hierarchical namespaces keep the data organized, which yields better storage and retrieval performance for an analytical use case and lowers the cost of analysis.
- Ingest
The ingestion phase identifies the technology and processes that are used to acquire the source data. This data can come from files, logs, and other types of unstructured data that must be put into the data lake.
The technology that is used will vary depending on the frequency that the data is transferred.- Batch movement of data, pipelines in Azure Synapse Analytics or Azure Data Factory
- Real-time ingestion of data, Apache Kafka for HDInsight or Stream Analytics .
- Store
The store phase identifies where the ingested data should be placed. Azure Data Lake Storage Gen2 provides a secure and scalable storage solution that is compatible with commonly used big data processing technologies. - Prep and train
The prep and train phase identifies the technologies that are used to perform data preparation and model training and scoring for machine learning solutions.
Common technologies that are used in this phase are Azure Synapse Analytics, Azure Databricks, Azure HDInsight, and Azure Machine Learning. - Model and serve
Involves the technologies that will present the data to users. These technologies can include visualization tools such as Microsoft Power BI, or analytical data stores such as Azure Synapse Analytics. Often, a combination of multiple technologies will be used depending on the business requirements.
- Big data processing and analytics
Usually refer to analytical workloads that involve massive volumes of data in a variety of formats that needs to be processed at a fast velocity - the so-called "three v's".
Big data services such as Azure Synapse Analytics, Azure Databricks, and Azure HDInsight can apply data processing frameworks such as Apache Spark, Hive, and Hadoop. - Data warehousing
Integrate large volumes of data stored as files in a data lake with relational tables in a data warehouse.
There are multiple ways to implement this kind of data warehousing architecture. The diagram shows a solution in whichAzure Synapse Analytics hosts pipelines to perform extract, transform, and load (ETL) processes using Azure Data Factory technology. - Real-time data analytics
streaming data requires a solution that can capture and process a boundless stream of data events as they occur.
Streaming events are often captured in a queue for processing. There are multiple technologies you can use to perform this task, including Azure Event Hubs as shown in the image.
Azure Stream Analytics enables you to create jobs that query and aggregate event data as it arrives, and write the results in an output sink - Data science and machine learning
Involves the statistical analysis of large volumes of data, often using tools such as Apache Spark and scripting languages such as Python. Azure Data Lake Storage Gen 2 provides a highly scalable cloud-based data store for the volumes of data required in data science workloads.
Analytical technique that organizations commonly use:
- Descriptive analytics
which answers the question “What is happening in my business?”. The data to answer this question is typically answered through the creation of a data warehouse in which historical data is persisted in relational tables for multidimensional modeling and reporting. - Diagnostic analytics,
which deals with answering the question “Why is it happening?”. This may involve exploring information that already exists in a data warehouse, but typically involves a wider search of your data estate to find more data to support this type of analysis. - Predictive analytics,
which enables you to answer the question “What is likely to happen in the future based on previous trends and patterns?” - Prescriptive analytics,
which enables autonomous decision making based on real-time or near real-time analysis of data, using predictive analytics.
To support the analytics needs of today's organizations, Azure Synapse Analytics combines a centralized service for data storage and processing with an extensible architecture through which linked services enable you to integrate commonly used data stores, processing platforms, and visualization tools.
A Synapse Analytics workspace defines an instance of the Synapse Analytics service in which you can manage the services and data resources needed for your analytics solution.
A workspace typically has a default data lake, which isimplemented as a linked serviceto an Azure Data Lake Storage Gen2 container.
Azure Synapse Analytics includes built-in support for creating, running, and managing pipelines that orchestrate the activities necessary
- to retrieve data from a range of sources,
- transform the data as required, and
- load the resulting transformed data into an analytical store.
- A built-in serverless pool that is optimized for using relational SQL semantics to query file-based data in a data lake.
use the built-in serverless pool for cost-effective analysis and processing of file data in the data lake - Customdedicated SQL pools that host relational data warehouses.
use dedicated SQL pools to create relational data warehouses for enterprise data modeling and reporting.
In Azure Synapse Analytics, you can create one or more Spark pools and use interactive notebooks to combine code and notes as you build solutions for data analytics, machine learning, and data visualization.
Exploring data with Data Explorer
Data Explorer uses an intuitive query syntax named Kusto Query Language (KQL) to enable high performance, low-latency analysis of batch and streaming data.
Azure Synapse Analytics can be integrated with other Azure data services for end-to-end analytics solutions. Integrated solutions include:
- Azure Synapse Link
enables near-realtime synchronization between operational data in Azure Cosmos DB, Azure SQL Database, SQL Server, and Microsoft Power Platform Dataverse and analytical data storage that can be queried in Azure Synapse Analytics. - Microsoft Power BI integration
enables data analysts to integrate a Power BI workspace into a Synapse workspace, and perform interactive data visualization in Azure Synapse Studio. - Microsoft Purview integration
enables organizations to catalog data assets in Azure Synapse Analytics, and makes it easier for data engineers to find data assets and track data lineage when implementing data pipelines that ingest data into Azure Synapse Analytics. - Azure Machine Learning integration
enables data analysts and data scientists to integrate predictive model training and consumption into analytical solutions.
- Large-scale data warehousing
Data warehousing includes the need to integrate all data, including big data, to reason over data for analytics and reporting purposes from a descriptive analytics perspective, independent of its location or structure. - Advanced analytics
Enables organizations to perform predictive analytics using both the native features of Azure Synapse Analytics, and integrating with other technologies such as Azure Machine Learning. - Data exploration and discovery
The serverless SQL pool functionality provided by Azure Synapse Analytics enables Data Analysts, Data Engineers and Data Scientist alike to explore the data within your data estate. This capability supports data discovery, diagnostic analytics, and exploratory data analysis. - Real time analytics
Azure Synapse Analytics can capture, store and analyze data in real-time or near-real time with features such as Azure Synapse Link, or through the integration of services such as Azure Stream Analytics and Azure Data Explorer. - Data integration
Azure Synapse Pipelines enables you to ingest, prepare, model and serve the data to be used by downstream systems. This can be used by components of Azure Synapse Analytics exclusively. - Integrated analytics
With the variety of analytics that can be performed on the data at your disposal, putting together the services in a cohesive solution can be a complex operation. Azure Synapse Analytics removes this complexity by integrating the analytics landscape into one service. That way you can spend more time working with the data to bring business benefit, than spending much of your time provisioning and maintaining multiple systems to achieve the same outcomes.
Python environment
1/3/2021
Study notes
Build a python app in isolation.
Solution: virtual environment
Self-contained copy of everything needed to run your program. This includes the Python interpreter and any libraries your program needs. By using a virtual environment, you can ensure your program will have access to the correct versions and resources to run correctly.
Steps:
- Create a virtual environment that won't affect the rest of the machine.
- Step into the virtual environment, where you specify the Python version and libraries that you need.
- Develop your program.
Create & go to app folder.
Run:
python -m venv env
Result a folders structure (may vary)
/env
|__/Include
|__/Lib
|_____site-packages
|__ /Scripts
Don't put your program files in the env directory. We suggest that you put your files in the srcdirectory or similar.
1.1. Activate the virtual environment
activate
#or
PATH_TO_ACTIVATE/activat
Result
(env) in front of prompt
To desactivateL
deactivate
ResultNo more (env) in front of prompt
1.2. Install a package
pip install python-dateutil
Once installed you will see
/
|__/env
|__/Lib
|__/dateutil
To see what packages are now installed in your virtual environmen
Run
pip freeze
Result:python-dateutil==2.8.2
...
More ways to install a package:
- Have a set of files on your machine and install from that source
cd <to where the package is on your machine>
python3 -m pip install . - Install from a GitHub repository
git+https://github.com/your-repo.git - Install from a compressed archive file
python3 -m pip install package.tar.gz
from datetime import *
from dateutil.relativedelta import *
now = datetime.now()
print(now)
now = now + relativedelta(months=1, weeks=1, hour=10)
print(now)
Distribute project to others for collaboration.
Create a project file
pip freeze > requirements.txt
Result:Creates a requirements.txtfile with all the packages that the program needs.
Create a .gitignorefile, and check in your application code and requirements.txt.
src/
requirements.txt
Check in the code to GitHub.Commit
Publish to GitHub
Check Online
Result - all files from requirements.txt are online (GitHub)
Consume a project
Create a folder
Go into
Get the git URL from GitHub, and run:
git clonehttps://github.com/corifeanu/Azure.git
Install requirements.
Run:
pip install -r requirements.txt
Update/Run your all (what is in src).
Manage dependencies
Kee upgrading your packages:
- Bug fixes.
The library that you use might have problems. For example, a feature doesn't work as intended and the author goes in to fix it. You most likely want to upgrade the package as soon as such a fix is in place. - Security issues.
Your package might have a security vulnerability. After such a fix is released, you want to upgrade the package to protect your company and your customers. - Additional features.
The release of more features is nice, though it isn't a strong reason to upgrade the package. Still, if there's a feature that you've been waiting for, you might want to upgrade for that reason
Specific version
pip install python-dateutil===1.4
Check if version is availablepip install python-dateutil===randomwords
Result:ERROR: Could not find a version that satisfies the requirement python-dateutil===randomwords (from versions: 1.4, 1.4.1, 1.5, 2.1, 2.2, 2.3, 2.4.0, 2.4.1, 2.4.1.post1, 2.4.2, 2.5.0, 2.5.1, 2.5.2, 2.5.3, 2.6.0, 2.6.1, 2.7.0, 2.7.1, 2.7.2, 2.7.3, 2.7.4, 2.7.5, 2.8.0, 2.8.1, 2.8.2)
ERROR: No matching distribution found for python-dateutil===randomwords
Use:
pip freeze
Result...
python-dateutil==1.4
...
Install last one
pip installpython-dateutil===2.8.2
Upgrade
pip install python-dateutil --upgrade
Versioning plain English:
- The leftmost number is called Major. If this number increases, many things have changed, and you can no longer assume that methods are named the same or have the same number of arguments as before.
- The middle number is called Minor. If it changes, a feature has been added.
- The rightmost number is called Patch. If this number increases, it most likely means that a bug has been corrected.
pip install "python-dateutil==2.8.*" --upgrade
Remove one package only:
pip uninstall python-dateutil
Remove all installed packages, by first writing them to a requirements.txt list and then removing all packages in that list.
pip freeze > requirements.txt
pip uninstall -r requirements.txt -y
Now if you run pip freeze, you see that it contains only the following output:
pip-autoremove==0.10.0
Language Understanding solution with Cognitive Services
27/2/2021
Study notes
Language Understanding solution means to use Natural language processing (NLP) and that is creating models able to interpret the semantic meaningof written or spoken language.
Language Service provides various features for understanding human language.
Language service features:
- Language detection - pre-configured
- Sentiment Analysis and opinion mining pre-configured
- Named Entity Recognition (NER) - pre-configured
- Personally identifying (PII) and health (PHI) information detection -pre-configured
- Summarization - pre-configured
- Key phrase extraction - pre-configured - pre-configured
- Entity linking - pre-configured
- Text analytics for health - pre-configured
- Custom text classification - learned
- Orchestration workflow
- Question answering - learned
- entity recognition ?
- intent recognition ?
- text classification ?
- Conversational language understanding - require a model to be built for prediction - learned
Core custom features
Helps users to build custom natural language understanding models to predict overall intent and extract important information from incoming utterances. - Custom named entity recognition (Custom NER) - require a model to be built for prediction- learned
Core custom features
- Pre-configured features
- Learned features. Learned features require building and training a model to correctly predict appropriate labels.
Building model:
All start with creating a Language Resource (you may use Cognitive service as well)
- Azure portal
- New resource -> Language resource
- Create (you will have keys and end point, like usual)
Select resource, give a name and select FREE price tier (5k transactions in 30 days)Once done, go to Keys and Endpoints
1. Create model via REST API- create your project
- import data
- train model
- deploy model
- query model
- Submit a request to the appropriate URI
- Send a request to get the status of that job. You will get it in JSON format.
{
"jobId":"{JOB-ID}",
"createdDateTime":"String",
"lastUpdatedDateTime":"String",
"expirationDateTime":"String",
"status":"running"
}
Ocp-Apim-Subscription-Key -> language resource key (see above)
2. Create model usingLangauge Studio
- Open Language Studio
Language Studio - Microsoft Azure - Select Directory, subscription and Resource type (language) and Resource (the one you just created above)
- From Create New, select what project you wish to create and go through the process.
Does not matter how you create and train the model, the final step is to query the model.
To query your model for a prediction, create a POST request to the appropriate URL with the appropriate body specified.
Authentication is necessary for any request you send (via header)
Body is JSON format.
Result is in JSON format
Example to detect language:
{
"kind": "LanguageDetection",
"parameters": {
"modelVersion": "latest"
},
"analysisInput":{
"documents":[
{
"id":"1",
"text": "This is a document written in English."
}
]
}
}
Result:
{
"kind": "LanguageDetectionResults",
"results": {
"documents": [{
"id": "1",
"detectedLanguage": {
"name": "English",
"iso6391Name": "en",
"confidenceScore": 1.0
},
"warnings": []
}],
"errors": [],
"modelVersion": "String"
}
}
Terms in NLP
- Utterance
The user input. Phrase that a user enters when interacting with the model (via an app that uses your Language Understanding model). - Intent
Task or action the user wants to perform (meaning of an utterance).
That something you want model to do and model has to understand and 'execute'
There is "None" intend - default, when there is no action defined for what you ask. - Entity
Add specific context to intents.
Steps:
- Define intents model must support.
- Define all possible utterance for every single intent (all possible a user may input to request an intend)
- Capture multiple different examples, or alternative ways of saying the same thing
- Vary the length of the utterances from short, to medium, to long
- Vary the location of the noun or subject of the utterance. Place it at the beginning, the end, or somewhere in between
- Use correct grammar and incorrect grammar in different utterances to offer good training data examples
- The precision, consistency and completeness of your labeled data are key factors to determining model performance.
- Precisely: Label each entity to its right type always. Only include what you want extracted, avoid unnecessary data in your labels.
- Consistently: The same entity should have the same label across all the utterances.
- Completely: Label all the instances of the entity in all your utterances.
- Add specific context to utterances using entities.
Example: Utterance: What time is it in London? / Intent: GetTime/ Entities: Location (London)- Learned entities
Flexible, use them in most cases. You define a learned component with a suitable name, and then associate words or phrases with it in training utterances.
When you train your model, it learns to match the appropriate elements in the utterances with the entity. - List entities
Useful when you need an entity with a specific set of possible values - for example, days of the week.
Ex: DayOfWeek entity that includes the values "Sunday", "Monday", etc - Prebuilt entities
Ueful for common types such as numbers, datetimes, and names.
Ex: when prebuilt components are added, you will automatically detect values such as "6" or organizations such as "Microsoft".
Let the Language service automatically detect the specified type of entity, and not have to train your model with examples of that entity
You can have up to five prebuilt components per entity.
- Learned entities
- Train it to learn intents and entities from sample utterances.
- Test it interactively or using a testing dataset with known labels
- Deploy a trained model to a public endpoint (users can use it)
- Review predictions and iterate on utterances to train your model
To use model in application you must publish a Language Understanding app for the created model.
Any app need an endpointused to query a specific feature varies.
https://{ENDPOINT}/text/analytics/{VERSION}/{FEATURE}
- {ENDPOINT}
The endpoint for authenticating your API request. For example, myLanguageService.cognitiveservices.azure.com - {VERSION}
The version number of the service you want to call. For example, v3.0 - {FEATURE}
The feature you're submitting the query to. For example, keyPhrases for key phrase detection
The Language service provides certain features without any model labeling or training.
Once you create your resource, you can send your data and use the returned results within your app. See Language service features(top this page)
Learned features
Require you to label data, train, and deploy your model to make it available to use in your application. See Language service features(top this page)
Consume a model via an App (Process predictions)
- Use REST APIs
- A programming language-specific SDKs.
- kind
Indicates language feature you're requesting.
Ex: Conversationfor conversational language understanding, EntityRecognition to detect entities. - parameters
Indicates the values for various input parameters.
Ex: projectName and deploymentName are required for conversational language understanding - analysis input
Specifies the input documents or text strings to be analyzed by the Language service.
The prediction results include the query utterance, the top (most likely) intent, other potential intents with their respective confidence score, and the entities that were detected.
Each entity includes a category and subcategory (when applicable) in addition to its confidence score (for example, "Edinburgh", which was detected as a location with confidence of 1.0).
The results may also include any other intents that were identified as a possible match, and details about the location of each entity in the utterance string.
The Language Understanding service can be deployed as a container, running in a local Dockerhost, an Azure Container Instance (ACI), or in an Azure Kubernetes Service (AKS) cluster.
Process:
- Container imagefor the specific Cognitive Services API you want to use is downloaded and deployed to a container host(local Docker server, ACI or AKS)
Ex. Get image:docker pull mcr.microsoft.com/azure-cognitive-services/textanalytics/language:latest - Client applications submit data to the endpointprovided by the containerized service, and retrieve resultsjust as they would from a Cognitive Services cloud resource in Azure.
Ex. run container
docker run --rm -it -p 5000:5000
--memory 4g
--cpus 1
mcr.microsoft.com/azure-cognitive-services/textanalytics/language
Eula=accept
Billing={ENDPOINT}
ApiKey={API_KEY}
Ex. Query service
http://localhost:5000/ - Periodically, usage metricsfor the containerized service are sent to a Cognitive Services resource in Azure in order to calculate billing for the service.
Text process and translate with Language Services
26/2/2021
Study notes
Language Service
- Language detection
Works with documents (max 5210 characters) and single phrases. Max 1000 items(ids) per collections.
For mixed languages => return predominantlanguage.
If not able to provide response return NaN or Not a Number. - Key phrase extraction
Process of evaluating the text of a document, or documents, and then identifying the main points around the context of the document
Max 5120 characters. Bigger document, better results. - Sentiment analysis
Quantifying positive / negative "message"
Return overall document sentiment and individual sentence sentiment for each document submitted to the service.
Overall document sentiment is based on sentences:- All sentences are neutral, the overall sentiment is neutral.
- Sentence classifications include only positive and neutral, the overall sentiment is positive.
- Sentence classifications include only negative and neutral, the overall sentiment is negative.
- Sentence classifications include positive and negative, the overall sentiment is mixed.
- Named entity recognition
Recognize entities mentioned in text (people, locations, time periods, organizations, etc)- Person
- Location
- DateTime
- Organization
- Address
- URL
- Entity linking
Entities reference links to Wikipedia articles.
Possibly, the same name might be applicable to more than one entity.
Wikipedia provides the knowledge base for the Text Analytics service. Specific article links are determined based on entity context within the text.
- Language detection
Compared with Language detection from Language service,this return if translation and transliteration is supported.
Return example.
[
{
"isTranslationSupported": true,
"isTransliterationSupported": true,
"language": "ja",
"score": 1.0
}
] - Translation and One-to-many translation
Specifying a single from parameterto indicate the source language,and one or more to parameters to specify the languages into which you want the text translated
[
{"translations":
[
{"text": "Hello", "to": "en"},
{"text": "Bonjour", "to": "fr"}
]
}
] - Script transliteration
Converting text from its native script to an alternativescript.
Translate to the same language but a different dialect.
- Translation options
Word alignment - spaces not always are used to separate words. - Sentence length - useful to know the length of a translation
- Profanity filtering
- NoAction: Profanities are translated along with the rest of the text.
- Deleted: Profanities are omitted in the translation.
- Marked: Profanities are indicated using the technique indicated in the profanityMarker parameter (if supplied).
Solution for businesses or industries in that have specific vocabularies of terms that require custom translation.
To solve this problem, you can create a custom model that maps your own sets of source and target terms for translation.
Use Azure Custom Translator Portal
Process is the same like Azure ML Designer. To create a custom model:- Create a workspacelinked to your Translator resource
- Create a project
- Upload training data files
- Traina model
How to use any of above services:
- Provision a resource (Language or Cognitive service)
- Call it using Endpoint and Key, submit data using JSON to REST interface or using an SDK.
Monitor Cognitive Service
25/2/2021
Study notes
1.Before do anything you must estimate the cost.
- Start here: Pricing Calculator | Microsoft Azure
- Search and select Azure Cognitive Services
- Create an estimate.
- To create an estimate that includes multiple Cognitive Services APIs, add additional Azure Cognitive Services products to the estimate.
- Azure Portal
- Subscriptions and select one subscription
- Go to Cost analysis (left)
3.Last but most important is create alerts (can be any area not only costs)
Checking from time to time is not an option.
Microsoft Azure provides alerting support for resources through the creation of alert rules.
There are two areas: Alert & Metrics.
Alerts
- Azure Portal
- Select Resource (Cognitive services in this case)
- Select Alerts (left)
- Create (top) ->Alert rule
- Scope is already selected because 'Azure Cognitive services' is set already. You can change.
- Condition on which the alert is triggered.
Any of three selectors can be used to narrow search. Let's look for 'key' and set that 'Regenerating keys' will trigger the alert - Optional actions, such as sending an email to an administrator notifying them of the alert, or running an Azure Logic App to address the issue automatically.
- Alert rule details, such as a name for the alert rule and the resource group in which it should be defined.
After creation, based on what you have you may get this.
- Azure Portal
- Select Resource (Cognitive services in this case)
- Select Metrics (left)
Selecting Data In / Azure cognitive services:
- add multiple metrics and aggregation type (sum, mean etc.)
- select chart type (bar, area chart, grid)
- drill into other services/logs
- create alert.
- Save to (Pin to dashboard: private or shared)
4. Diagnostic logging enables you to capture rich operational data for a Cognitive Services resource, which can be used to analyze service usage and troubleshoot problems.
Create destination to log data (where to store data coming from logs)
You can use Azure Event Hub as a destination in order to then forward the data on to a custom telemetry solution, and you can connect directly to some third-party solutions.
Does not matter what you do you need one or both resources (must create them before configuring diagnostic logging.)
- Azure Log Analytics- a service that enables you to query and visualize log data within the Azure portal.
- Azure Storage - a cloud-based data store that you can use to store log archives (which can be exported for analysis in other tools as needed).
Secure cognitive services in Azure
25/2/2021
Study notes
By default, access to cognitive services resources is restricted by using subscription keys. Management of access to these keys is a primary consideration for security.
You should regenerate keys regularly to protect against the risk of keys being shared with or accessed by unauthorized users. You can regenerate keys by using the visual interface in the Azure portal, or by using theaz cognitiveservices account keys regenerate Azure command-line interface (CLI) command.
Azure Key Vault
Keys and other secrets mut not be kept in configuration files.Always use Azure Key Vault.
Access to the key vault is granted to security principals, which you can think of user identities that are authenticated using Azure AD.
- Administrators assign a security principal to an application (in which case it is known as a service principal) to define a managed identity for the application.
- The application can then use this identity to access the key vault and retrieve a secret to which it has access.
Most common REST interface require token-based authentication.
- You should have a subscription key (initial request)
- Obtain an authentication token (valid 10 minutes)
- Present the token to validate that the caller has been authenticated.
Some Cognitive Services support Azure Active Directory authentication, enabling you to grant access to specific service principals or managed identities for apps and services running in Azure.
Network Security (network access restrictions)
- In Azure portal
- Cognitive service - you have created
- Networking
- Set restrictions
Example:
Secure key access with Azure Key Vault
You can develop applications that consume cognitive services by using a key for authentication.
This means that the application code must be able to obtain the key.
- Store the key in an environment variable or a configuration file where the application is deployed, not good.
- Store the key securely in Azure Key Vault, and provide access to the key through a managed identity(a user account used by the application itself).
Steps:
Create Resource: Key Vault and store Cognitive Service Key1 in it, Name must be Cognitive-Services-Key
Create a Service Principal (we need resource name and subscription Id)
Find its (Service Principal) Object Id (We need Service Principal appId)
Grand access to Service Principal to secret (which is the Key1 of Cognitive service, we need Key Vault name, Service Principal Object Id and possible Resource Group name)
Now we're ready to use the service principal identity in an application,
It can access the secret cognitive services key in your key vault and use it to connect to your cognitive services resource.
Python development environment for Azure Machine Learning
18/2/2021
Set up a Python development environment for Azure Machine Learning
Local and Data Science Virtual Machine (DSVM)
You can do it manually
Create folder
.azureml
Create file
.azureml/config.js
{
"subscription_id": "SUBSCRIPTION_ID",
"resource_group": "vscode-ml-rg",
"workspace_name": "vscode-ml-ws"
}
Or from Azure ML Studio
Download config file.
You can load a specific config.json file
You can load a workspace from current run context.
Applicable if you are on a compute cluster.
References:
Set up Python development environment - Azure Machine Learning | Microsoft Learn
azureml.core.run.Run class - Azure Machine Learning Python | Microsoft Learn
Local and Data Science Virtual Machine (DSVM)
# Create Resource group and Workspace
from azureml.core import Workspace
ws = Workspace.create(name='vscodeml-ws',
subscription_id='SUBSCRIPTION_ID',
resource_group='vscodeml-rg',
create_resource_group=True,
location='eastus2'
)
subscription_id='SUBSCRIPTION_ID',
resource_group='vscodeml-rg',
create_resource_group=True,
location='eastus2'
)
# Write config.json file file (confg file for the environent)
# A folder .azureml will be created and the config.json file will be created into it
ws.write_config(path="./", file_name="config.json")
Create folder
.azureml
Create file
.azureml/config.js
{
"subscription_id": "SUBSCRIPTION_ID",
"resource_group": "vscode-ml-rg",
"workspace_name": "vscode-ml-ws"
}
Or from Azure ML Studio
Download config file.
# Finally load the workspace (interactive login)
ws = Workspace.from_config()
# print details
ws.get_details()
# or just print short messageprint('Ready to use Azure ML {} to work with {}'.format(azureml.core.VERSION, ws.name))
You can load a specific config.json file
ws = Workspace.from_config(path="my/path/config.json")
You can load a workspace from current run context.
Applicable if you are on a compute cluster.
# did not used it in production yet
Run.get_context().experiment.workspace
Run.get_context().experiment.workspace
References:
Set up Python development environment - Azure Machine Learning | Microsoft Learn
azureml.core.run.Run class - Azure Machine Learning Python | Microsoft Learn
Run a Job using hyperparameter (tune job)