Accessing Salesforce or Databricks for automated reports in Posit Connect?

Our new blog explores Service Account OAuth integrations: secure, static machine identity for streamlined data access.

Learn more here: https://posit.co/blog/using-service-account-identities-posit-connect/

Pin your data and model objects to Databricks Volumes from your #RStats script!

With the pins package, you can publish data, models, and other objects for sharing across projects and with your colleagues. You “pin” an object to a board, like Dropbox, Posit Connect, or Amazon S3.

Now, we have merged support for pinning objects to #Databricks Volumes with the `board_databricks()` function!

Read more about this integration in our latest blog post: https://posit.co/blog/pins-in-databricks/

We are thrilled to announce #RStudio IDE and Posit Workbench 2024.09.0!

• Bundled #Quarto v1.5
• Managed credential support for #AWS, #Azure, #Databricks, and #Snowflake
• The “Save as Image” dialog now
controls whether plots are scaled according to the current display’s DPI

Learn more in the blog post: https://posit.co/blog/rstudio-2024-09-0-whats-new/

The sparklyr package and friends have been getting some important updates in the past few months!

sparklyr is a package that allows you to interact with Spark using familiar R interfaces, such as dplyr, broom, and DBI. You can also gain access to Spark's distributed Machine Learning libraries, Structure Streaming, and ML Pipelines from R.

Read more in the blog post: https://blogs.rstudio.com/ai/posts/2024-04-22-sparklyr-updates/

I took a cold brew mocha canned beverage at the #databricks bellevue office and I cannot sleep. That company knows how to keep their employees awake.

Btw, the event was organized by #pydata #python #seattle

Figured I should do an #introduction! I'm Peter, a queer dude living in Vienna. Daytime I work for the Austrian Railways #oebb as data engineer, ETL, #devops, #databricks. Love #Python, learning #rust. I'm an avid "Alltagsradler" (sb. who runs daily errands by bicycle) and proponent of making our cities more #walkable and safe for everyone. Recently, I started a blog over at https://pyter.at. Nice to be here!

Did someone say enhancements for #Databricks users in Posit Workbench?

The recent release of Posit Workbench includes two specific enhancements for Databricks users:

Delegated Databricks credentials

New Databricks UI within #RStudio

With this release, users can log in to a given Databricks workspace when they start an RStudio or #VSCode session and interact directly with the clusters in that Workspace from their preferred environment.

Learn more: https://posit.co/blog/posit-databricks-webinar-dec2023-q-a/

We’re excited to announce the latest release of Posit Workbench and RStudio IDE 2023.12.0!

• GitHub Copilot now generally available
• #Quarto and R Markdown improvements
• New #Databricks Pane and Databricks ODBC Driver

Learn more in the blog post: https://posit.co/blog/rstudio-2023-12-0-whats-new/

The new version of pysparklyr is on CRAN!

Pysparklyr is the new extension to sparklyr that allows you to interact with #Spark & Databricks Connect. The new version has big user-facing updates that make working with #Databricks and #RStats together even easier.

Read more: https://posit.co/blog/pysparklyr-for-interacting-with-spark-databricks-connect/

We are thrilled to announce that the latest version of sparklyr is on CRAN. sparklyr is the popular and powerful #RStats interface for #Apache #Spark, including Spark clusters hosted in #Databricks.

Thanks to the new Spark Connect protocol, you can access Spark’s powerful distributed computing features from RStudio Desktop, a Posit Workbench instance, or any running R terminal or process.

Learn more in the blog post: https://posit.co/blog/databricks-clusters-in-rstudio-with-sparklyr/

New blog post: How to integrate Kedro and Databricks Connect

In this blog post, our colleague Diego Lira explains how to use Databricks Connect with Kedro for a development experience that works completely inside an IDE.

https://kedro.org/blog/how-to-integrate-kedro-and-databricks-connect

Install it with

```
pip install databricks-connect
```

New blog post: How to use Databricks managed Delta tables in a Kedro project

In this post our colleague Jannic Holzer explains how to use a newly-released dataset for managed Delta tables in Databricks within your Kedro project.

https://kedro.org/blog/managed-delta-tables-kedro-dataset

Install it with

```
pip install "kedro-datasets[databricks.ManagedTableDataSet]"
```

Ever wanted to write #ETL pipelines in bare code - with little thought ?

With #DeltaLiveTables, the latest ETL framework from #Databricks, you can leverage the power of code to create reliable, scalable, and easy-to-maintain #DataPipelines . Join me as I explore the framework and share my experiences with it.

https://www.fabianstadler.com/2023/04/data-pipelines-as-code-with-delta-live-tables.html

Yesterday we tried to upload 10M rows to an #Azure Table using #ApacheSpark and #Databricks. We hit 333K rows per minute at our best.

Wondering if anyone here has done something similar?

From what I read, the max transactions per second on an Azure Table is 20K so… I guess we can try to speed it up a bit further.

Anyone 1) running #Posit #RStudio Server on #Azure #Databricks? Or perhaps 2) connecting to Databricks from RStudio Server running somewhere else?

Curious to hear about your setup and how well it works.

Apparently #1 requires that you disable credential passthrough, which IT is saying is a nonstarter. And #2 requires RStudio Server, which is Linux-only. But IT won't let us use Linux--we can only use a Windows VDI.

Question to all the #DataEngineer s out there: what conferences do you recommend in the field? I prefer ones hosted in Europe so I don’t have to travel much but I am open to other places. In my team people join #aws or #databricks conferences/summit and I also love something like #europython (although not specific to data engineering). Any suggestions?

Hi #hachyderm! My #introduction:

I'm a data engineer in #Austin #Texas, where I live with my partner and her cat.

I work mostly with #azure, #python, #databricks, and #sql. Learning more about #machinelearning and #dataviz.

A couple of things that are interesting to me now:

* What kind of drift/movement exists in ESG scores over time?
* What kind of variations in financial portfolio composition exist in different parts of the USA?

Other interests: nature, travel, family, and friends

An #introduction…

My name is Alejandra Hernandez-Segura and I am a #mexican data engineer living in #TheNetherlands I am a #python and #rstats programmer and a user of #databricks and #aws

I am a member of #rladies and #pyladies One of my main goals is to help other women and other minorities, including all Latin American people enter this technical field that so far has little diversity!

I love reading , listening to podcasts, learning about history, #running and doing #yoga

I just did an #introduction a few days ago, but I've moved servers, so let's try one more time, for the cheap seats in the back!

I'm currently a data analyst/product #DataScientist working with free-to-play #VideoGames, and living in #Halifax, #NovaScotia, #Canada. I've done a lot of work on #Analytics design, with a focus on ensuring player telemetry events are sensibly cross-referenceable, and looking for relationships between engagement with different game features and business outcomes.

Business teams in freemium games love looking for magic buttons.

I primarily use #SQL, #Pandas, #Statsmodels, and #SKLearn on #Databricks (#Python), and #JuliaLang (DataFrames.jl, GLM.jl, Gadfly.jl, and Makie.jl, etc) for smaller, locally run projects. My interests lie in expanding the library of ML models I have in my back pocket for performing inference based knowledge generation. I'm not super keen on automating products with quasi-black-boxes for the sake of revenue optimization. If I'm not personally learning something new about people through my work, I don't usually see the value in it.

I did my BSc in #Physics and my MSc in #Astronomy, and, though I had dreams of progressing further down that pipeline, life kind of got in the way. Between the two degrees, I worked at the #Edmonton #Planetarium for four years as a presenter/operator (should out to the #ZeidlerDome at #TWoSE!).

I'm a life-long #Trekie, thanks to my mother. I grew up with #TNG and #DS9, and watched the first 5 seasons of #VOY before leaving home for university. Currently very bullish on #SNW and #LDS.

I'm also a lifelong #Baseball fan (#BlueJays and #Expos), and actively play rec #Softball.

About a year ago, I purchased my first lens-swappable digital #camera, and have been figuring out #Photography ever since. Most of my posts have focused on sharing my pictures, though I've recently decided to start a dedicated account for that on a #PixelFed server, for the sake of searchability.

My wife is currently studying political sociology, and I find her work fascinating. She's not currently on the Fediverse, but maybe one of these days.

This has really lost all sense of narrative flow, hasn't it? Oops!